Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
853.431215ms
FAIL (2025-10-06 23:17:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 853.452607ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
419.731031ms
FAIL (2025-10-06 23:17:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 419.749907ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X......X...X...] duration:
401.562672ms
FAIL (2025-10-06 23:17:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 401.611374ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................X..........XX..........X.....X...X...] duration:
535.760372ms
FAIL (2025-10-06 23:17:20 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
8s) [seg_pos={0 0}, upload=810623]: Begin upload segment...
Oct 6 23:14:35.231159887 (921h6m10.583944995s) [seg_pos={0 0}, upload=810623]: Done begin upload segment.
Oct 6 23:14:35.231177129 (921h6m10.583962237s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231202467 (921h6m10.583987575s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231445945 (921h6m10.584231042s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231464639 (921h6m10.584249747s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231662370 (921h6m10.584447468s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231680665 (921h6m10.584465773s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231888995 (921h6m10.584674093s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231977101 (921h6m10.584762198s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232134006 (921h6m10.584919114s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232215128 (921h6m10.585000236s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232343719 (921h6m10.585128817s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232451020 (921h6m10.585236118s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232590893 (921h6m10.585376001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232671273 (921h6m10.585456381s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232794845 (921h6m10.585579953s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.233021411 (921h6m10.585806519s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.301739905 (921h6m10.654525013s) [upload=810623]: Next returned nil segment
Oct 6 23:14:35.301751626 (921h6m10.654536734s) [upload=810623]: Waiting for error group managing segments...
Oct 6 23:14:35.308525018 (921h6m10.661310126s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309028252 (921h6m10.66181336s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309677310 (921h6m10.662462408s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309896092 (921h6m10.6626812s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310034271 (921h6m10.662819379s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310195242 (921h6m10.66298034s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:15:05.199595893 (921h6m40.552381001s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199636880 (921h6m40.552421988s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199832757 (921h6m40.552617885s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 5.161.89.252:53308->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199933757 (921h6m40.552718865s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:52568->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199996945 (921h6m40.552782053s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 5.161.89.252:8812->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200046488 (921h6m40.552831617s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200054824 (921h6m40.552839922s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:12616->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200091874 (921h6m40.552877001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 5.161.89.252:39316->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200106040 (921h6m40.552891148s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200139673 (921h6m40.552924771s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:18720->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200164760 (921h6m40.552949858s) [seg_pos={0 0}, upload=810623]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Oct 6 23:15:05.216712375 (921h6m40.569497483s) [upload=810623]: Done uploading segments...
Oct 6 23:15:05.216730419 (921h6m40.569515517s) [upload=810623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 535.804655ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
922.526569ms
FAIL (2025-10-06 23:17:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 922.545124ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X......X...X..] duration:
499.829173ms
FAIL (2025-10-06 23:17:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.852767ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
609.617877ms
FAIL (2025-10-06 23:16:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 609.634278ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
404.753228ms
FAIL (2025-10-06 23:16:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 404.771563ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................................X......X...X.] duration:
461.247982ms
FAIL (2025-10-06 23:16:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.27424ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................X..........XX..........X.....X...X..] duration:
524.967204ms
FAIL (2025-10-06 23:16:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
8s) [seg_pos={0 0}, upload=810623]: Begin upload segment...
Oct 6 23:14:35.231159887 (921h6m10.583944995s) [seg_pos={0 0}, upload=810623]: Done begin upload segment.
Oct 6 23:14:35.231177129 (921h6m10.583962237s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231202467 (921h6m10.583987575s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231445945 (921h6m10.584231042s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231464639 (921h6m10.584249747s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231662370 (921h6m10.584447468s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231680665 (921h6m10.584465773s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231888995 (921h6m10.584674093s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231977101 (921h6m10.584762198s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232134006 (921h6m10.584919114s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232215128 (921h6m10.585000236s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232343719 (921h6m10.585128817s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232451020 (921h6m10.585236118s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232590893 (921h6m10.585376001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232671273 (921h6m10.585456381s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232794845 (921h6m10.585579953s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.233021411 (921h6m10.585806519s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.301739905 (921h6m10.654525013s) [upload=810623]: Next returned nil segment
Oct 6 23:14:35.301751626 (921h6m10.654536734s) [upload=810623]: Waiting for error group managing segments...
Oct 6 23:14:35.308525018 (921h6m10.661310126s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309028252 (921h6m10.66181336s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309677310 (921h6m10.662462408s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309896092 (921h6m10.6626812s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310034271 (921h6m10.662819379s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310195242 (921h6m10.66298034s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:15:05.199595893 (921h6m40.552381001s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199636880 (921h6m40.552421988s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199832757 (921h6m40.552617885s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 5.161.89.252:53308->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199933757 (921h6m40.552718865s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:52568->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199996945 (921h6m40.552782053s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 5.161.89.252:8812->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200046488 (921h6m40.552831617s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200054824 (921h6m40.552839922s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:12616->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200091874 (921h6m40.552877001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 5.161.89.252:39316->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200106040 (921h6m40.552891148s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200139673 (921h6m40.552924771s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:18720->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200164760 (921h6m40.552949858s) [seg_pos={0 0}, upload=810623]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Oct 6 23:15:05.216712375 (921h6m40.569497483s) [upload=810623]: Done uploading segments...
Oct 6 23:15:05.216730419 (921h6m40.569515517s) [upload=810623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 524.991649ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
444.62744ms
FAIL (2025-10-06 23:16:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 444.648049ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................................X......X...X] duration:
30.001228033s
FAIL (2025-10-06 23:15:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001245636s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
43.81643726s
FAIL (2025-10-06 23:15:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 43.816455505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................X..........XX..........X.....X...X.] duration:
28.954291695s
FAIL (2025-10-06 23:15:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
8s) [seg_pos={0 0}, upload=810623]: Begin upload segment...
Oct 6 23:14:35.231159887 (921h6m10.583944995s) [seg_pos={0 0}, upload=810623]: Done begin upload segment.
Oct 6 23:14:35.231177129 (921h6m10.583962237s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231202467 (921h6m10.583987575s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231445945 (921h6m10.584231042s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231464639 (921h6m10.584249747s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231662370 (921h6m10.584447468s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231680665 (921h6m10.584465773s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231888995 (921h6m10.584674093s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231977101 (921h6m10.584762198s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232134006 (921h6m10.584919114s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232215128 (921h6m10.585000236s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232343719 (921h6m10.585128817s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232451020 (921h6m10.585236118s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232590893 (921h6m10.585376001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232671273 (921h6m10.585456381s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232794845 (921h6m10.585579953s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.233021411 (921h6m10.585806519s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.301739905 (921h6m10.654525013s) [upload=810623]: Next returned nil segment
Oct 6 23:14:35.301751626 (921h6m10.654536734s) [upload=810623]: Waiting for error group managing segments...
Oct 6 23:14:35.308525018 (921h6m10.661310126s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309028252 (921h6m10.66181336s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309677310 (921h6m10.662462408s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309896092 (921h6m10.6626812s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310034271 (921h6m10.662819379s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310195242 (921h6m10.66298034s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:15:05.199595893 (921h6m40.552381001s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199636880 (921h6m40.552421988s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199832757 (921h6m40.552617885s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 5.161.89.252:53308->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199933757 (921h6m40.552718865s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:52568->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199996945 (921h6m40.552782053s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 5.161.89.252:8812->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200046488 (921h6m40.552831617s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200054824 (921h6m40.552839922s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:12616->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200091874 (921h6m40.552877001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 5.161.89.252:39316->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200106040 (921h6m40.552891148s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200139673 (921h6m40.552924771s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:18720->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200164760 (921h6m40.552949858s) [seg_pos={0 0}, upload=810623]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Oct 6 23:15:05.216712375 (921h6m40.569497483s) [upload=810623]: Done uploading segments...
Oct 6 23:15:05.216730419 (921h6m40.569515517s) [upload=810623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 28.954310079s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
775.457589ms
FAIL (2025-10-06 23:15:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 775.480822ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................X..........XX..........X.....X...X] duration:
30.018583807s
FAIL (2025-10-06 23:15:05 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
8s) [seg_pos={0 0}, upload=810623]: Begin upload segment...
Oct 6 23:14:35.231159887 (921h6m10.583944995s) [seg_pos={0 0}, upload=810623]: Done begin upload segment.
Oct 6 23:14:35.231177129 (921h6m10.583962237s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231202467 (921h6m10.583987575s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231445945 (921h6m10.584231042s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231464639 (921h6m10.584249747s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231662370 (921h6m10.584447468s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231680665 (921h6m10.584465773s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231888995 (921h6m10.584674093s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.231977101 (921h6m10.584762198s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232134006 (921h6m10.584919114s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232215128 (921h6m10.585000236s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232343719 (921h6m10.585128817s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232451020 (921h6m10.585236118s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232590893 (921h6m10.585376001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232671273 (921h6m10.585456381s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.232794845 (921h6m10.585579953s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.233021411 (921h6m10.585806519s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Uploading piece...
Oct 6 23:14:35.301739905 (921h6m10.654525013s) [upload=810623]: Next returned nil segment
Oct 6 23:14:35.301751626 (921h6m10.654536734s) [upload=810623]: Waiting for error group managing segments...
Oct 6 23:14:35.308525018 (921h6m10.661310126s) [noise=true , address=145.79.227.5:4221 , piece_id=GX5MVSFGABSJDQLNPQUNBDCKRUGHPPTRJ2Y4KTSJM3GBHDVMWSUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309028252 (921h6m10.66181336s) [noise=true , address=145.79.227.13:4802 , piece_id=U6QBCPLFOT7FKZER4GTZHZONSLGI32ZFKQ6YMHLMXBVT6Q3G3Q5A, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309677310 (921h6m10.662462408s) [noise=true , address=145.79.227.14:4839 , piece_id=YGAV5O7IBCW654XHKAKGKQRQ4PIHOQYNPRKNNSI4LBZ3N2I24HDQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.309896092 (921h6m10.6626812s) [noise=true , address=145.79.227.3:4150 , piece_id=5JCRCAVAZMPJWDWBPIMMTXYESJQYHCBYZV2D5CGCAWDMD73HC26Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310034271 (921h6m10.662819379s) [noise=true , address=145.79.227.17:4970 , piece_id=EQDZAIB72F2JRHKJD444PCM4AG225NFL4BFT6L74E7QSFVHDH7QQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:14:35.310195242 (921h6m10.66298034s) [noise=true , address=145.79.227.2:4113 , piece_id=3B4UQO3A2V5SDEQTEPLDVFD2GFGIIDZI3GTOC4Q7B3MRMSUTFVUQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: <nil>
Oct 6 23:15:05.199595893 (921h6m40.552381001s) [noise=true , address=77.90.31.10:4517 , piece_id=73CMBGB7TGVXWGQC6IUNB54POXYMSHWYRRRV7GTXXZDF76LIAQVQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199636880 (921h6m40.552421988s) [noise=true , address=45.91.116.12:4225 , piece_id=WXDK5FN4PP4LPW3UF6ZLBXWLIZAXLP64BWNGY2DFIKGIFNESOFLQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199832757 (921h6m40.552617885s) [noise=true , address=77.90.31.21:4877 , piece_id=EUY5Q64A7NBYOTT4I5MTH3SZBEDXYJTI42W6VYOTNL4B5NMTDKNQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 5.161.89.252:53308->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199933757 (921h6m40.552718865s) [noise=true , address=77.90.31.18:4805 , piece_id=2B2PP4HY4ILPNQYUCJH3AZCLM2KQJHYV4QYN43OM5BJQFHM2SI4Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:52568->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.199996945 (921h6m40.552782053s) [noise=true , address=77.90.31.21:4879 , piece_id=YMGHKTOSRW6HQSEGA5Z74727YX5PI3LMGNJEU7JQDYL5UGJ2AHMQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 5.161.89.252:8812->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200046488 (921h6m40.552831617s) [noise=true , address=77.90.31.12:4592 , piece_id=IXYLGHN3BS4KSQVM7MJDL5JBR2NGXB3HBQPS62DGJKFEMJ5EUCYQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200054824 (921h6m40.552839922s) [noise=true , address=45.91.116.9:4262 , piece_id=LZITJ3U4QUE4VQM2NUAZMPJ5SBEBSVEQW427LRKG4NTRU7KKQUQQ, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:12616->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200091874 (921h6m40.552877001s) [noise=true , address=77.90.31.16:4735 , piece_id=VARS5MVTAXS2Y7C2IDCX7XW4Y72G574Y5SK3JN4LFDBDAJYEBU3Q, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 5.161.89.252:39316->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200106040 (921h6m40.552891148s) [noise=true , address=77.90.31.11:4554 , piece_id=BBSDAQ52W6Z26QH3XXEPJVW34ZSZCSFLREX44Q4TDSGOPXKA6ANA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200139673 (921h6m40.552924771s) [noise=true , address=77.90.31.13:4626 , piece_id=ANQSEX2GFRGK5AHO5IFQYKXB5ABZDICBTKNY4OKOP5DUZCQYS4QA, seg_pos={0 0}, upload=810623]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:18720->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:05.200164760 (921h6m40.552949858s) [seg_pos={0 0}, upload=810623]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Oct 6 23:15:05.216712375 (921h6m40.569497483s) [upload=810623]: Done uploading segments...
Oct 6 23:15:05.216730419 (921h6m40.569515517s) [upload=810623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.018600298s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
822.399052ms
FAIL (2025-10-06 23:14:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 822.420462ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................X..........XX..........X.....X...] duration:
551.946909ms
FAIL (2025-10-06 23:14:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:11:48.517381839 (921h3m23.870166937s) [upload=810581]: Starting upload
Oct 6 23:11:48.517430841 (921h3m23.870215949s) [upload=810581]: Done starting upload
Oct 6 23:11:48.517516051 (921h3m23.870301149s) [upload=810581]: Uploading segments...
Oct 6 23:11:48.517644803 (921h3m23.8704299s) [upload=810581]: Got next segment. Inline: false
Oct 6 23:11:48.517676692 (921h3m23.8704618s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:11:48.517680208 (921h3m23.870465317s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:11:48.538737567 (921h3m23.891522675s) [seg_pos={0 0}, upload=810581]: Begin upload segment...
Oct 6 23:11:48.538764999 (921h3m23.891550107s) [seg_pos={0 0}, upload=810581]: Done begin upload segment.
Oct 6 23:11:48.538783212 (921h3m23.89156832s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.538801406 (921h3m23.891586505s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539074769 (921h3m23.891859867s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539078877 (921h3m23.891863985s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539310673 (921h3m23.892095771s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539328626 (921h3m23.892113725s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539541786 (921h3m23.892326885s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539546105 (921h3m23.892331202s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539757621 (921h3m23.892542729s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539765105 (921h3m23.892550203s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540007210 (921h3m23.892792318s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540070559 (921h3m23.892855667s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540246559 (921h3m23.893031657s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540660275 (921h3m23.893445373s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.541324632 (921h3m23.89410974s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.542295153 (921h3m23.895080261s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.603736951 (921h3m23.956522059s) [upload=810581]: Next returned nil segment
Oct 6 23:11:48.603748833 (921h3m23.956533941s) [upload=810581]: Waiting for error group managing segments...
Oct 6 23:11:48.612397173 (921h3m23.965182271s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.612520434 (921h3m23.965305542s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613048776 (921h3m23.965833874s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613406328 (921h3m23.966191426s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.614573888 (921h3m23.967358987s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615075119 (921h3m23.967860227s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615308858 (921h3m23.968093966s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615443009 (921h3m23.968228107s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615976040 (921h3m23.968761148s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:12:18.505364225 (921h3m53.858149333s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505470433 (921h3m53.858255542s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505532189 (921h3m53.858340762s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: rpc: dial tcp 45.91.116.13:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505618622 (921h3m53.85840373s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:61408->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505657595 (921h3m53.858442703s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505699794 (921h3m53.858484892s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 5.161.89.252:38782->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505756741 (921h3m53.858541849s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 5.161.89.252:28690->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505786897 (921h3m53.858572005s) [seg_pos={0 0}, upload=810581]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:12:18.523379844 (921h3m53.876164942s) [upload=810581]: Done uploading segments...
Oct 6 23:12:18.523393139 (921h3m53.876178237s) [upload=810581]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 551.969641ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
575.576883ms
FAIL (2025-10-06 23:14:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 575.605946ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X......X...] duration:
515.431016ms
FAIL (2025-10-06 23:14:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 515.448006ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
767.425014ms
FAIL (2025-10-06 23:13:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 767.447756ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................X..........XX..........X.....X..] duration:
401.049479ms
FAIL (2025-10-06 23:13:35 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:11:48.517381839 (921h3m23.870166937s) [upload=810581]: Starting upload
Oct 6 23:11:48.517430841 (921h3m23.870215949s) [upload=810581]: Done starting upload
Oct 6 23:11:48.517516051 (921h3m23.870301149s) [upload=810581]: Uploading segments...
Oct 6 23:11:48.517644803 (921h3m23.8704299s) [upload=810581]: Got next segment. Inline: false
Oct 6 23:11:48.517676692 (921h3m23.8704618s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:11:48.517680208 (921h3m23.870465317s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:11:48.538737567 (921h3m23.891522675s) [seg_pos={0 0}, upload=810581]: Begin upload segment...
Oct 6 23:11:48.538764999 (921h3m23.891550107s) [seg_pos={0 0}, upload=810581]: Done begin upload segment.
Oct 6 23:11:48.538783212 (921h3m23.89156832s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.538801406 (921h3m23.891586505s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539074769 (921h3m23.891859867s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539078877 (921h3m23.891863985s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539310673 (921h3m23.892095771s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539328626 (921h3m23.892113725s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539541786 (921h3m23.892326885s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539546105 (921h3m23.892331202s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539757621 (921h3m23.892542729s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539765105 (921h3m23.892550203s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540007210 (921h3m23.892792318s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540070559 (921h3m23.892855667s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540246559 (921h3m23.893031657s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540660275 (921h3m23.893445373s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.541324632 (921h3m23.89410974s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.542295153 (921h3m23.895080261s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.603736951 (921h3m23.956522059s) [upload=810581]: Next returned nil segment
Oct 6 23:11:48.603748833 (921h3m23.956533941s) [upload=810581]: Waiting for error group managing segments...
Oct 6 23:11:48.612397173 (921h3m23.965182271s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.612520434 (921h3m23.965305542s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613048776 (921h3m23.965833874s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613406328 (921h3m23.966191426s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.614573888 (921h3m23.967358987s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615075119 (921h3m23.967860227s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615308858 (921h3m23.968093966s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615443009 (921h3m23.968228107s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615976040 (921h3m23.968761148s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:12:18.505364225 (921h3m53.858149333s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505470433 (921h3m53.858255542s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505532189 (921h3m53.858340762s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: rpc: dial tcp 45.91.116.13:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505618622 (921h3m53.85840373s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:61408->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505657595 (921h3m53.858442703s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505699794 (921h3m53.858484892s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 5.161.89.252:38782->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505756741 (921h3m53.858541849s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 5.161.89.252:28690->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505786897 (921h3m53.858572005s) [seg_pos={0 0}, upload=810581]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:12:18.523379844 (921h3m53.876164942s) [upload=810581]: Done uploading segments...
Oct 6 23:12:18.523393139 (921h3m53.876178237s) [upload=810581]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 401.066521ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
4.264574239s
FAIL (2025-10-06 23:13:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.264592995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
394.669406ms
FAIL (2025-10-06 23:13:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 394.686738ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X......X..] duration:
659.413828ms
FAIL (2025-10-06 23:13:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 659.465746ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.263604554s
FAIL (2025-10-06 23:12:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.263616306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
38.238540238s
FAIL (2025-10-06 23:12:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.238558401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................X..........XX..........X.....X.] duration:
1.506073609s
FAIL (2025-10-06 23:12:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:11:48.517381839 (921h3m23.870166937s) [upload=810581]: Starting upload
Oct 6 23:11:48.517430841 (921h3m23.870215949s) [upload=810581]: Done starting upload
Oct 6 23:11:48.517516051 (921h3m23.870301149s) [upload=810581]: Uploading segments...
Oct 6 23:11:48.517644803 (921h3m23.8704299s) [upload=810581]: Got next segment. Inline: false
Oct 6 23:11:48.517676692 (921h3m23.8704618s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:11:48.517680208 (921h3m23.870465317s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:11:48.538737567 (921h3m23.891522675s) [seg_pos={0 0}, upload=810581]: Begin upload segment...
Oct 6 23:11:48.538764999 (921h3m23.891550107s) [seg_pos={0 0}, upload=810581]: Done begin upload segment.
Oct 6 23:11:48.538783212 (921h3m23.89156832s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.538801406 (921h3m23.891586505s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539074769 (921h3m23.891859867s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539078877 (921h3m23.891863985s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539310673 (921h3m23.892095771s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539328626 (921h3m23.892113725s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539541786 (921h3m23.892326885s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539546105 (921h3m23.892331202s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539757621 (921h3m23.892542729s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539765105 (921h3m23.892550203s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540007210 (921h3m23.892792318s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540070559 (921h3m23.892855667s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540246559 (921h3m23.893031657s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540660275 (921h3m23.893445373s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.541324632 (921h3m23.89410974s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.542295153 (921h3m23.895080261s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.603736951 (921h3m23.956522059s) [upload=810581]: Next returned nil segment
Oct 6 23:11:48.603748833 (921h3m23.956533941s) [upload=810581]: Waiting for error group managing segments...
Oct 6 23:11:48.612397173 (921h3m23.965182271s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.612520434 (921h3m23.965305542s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613048776 (921h3m23.965833874s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613406328 (921h3m23.966191426s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.614573888 (921h3m23.967358987s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615075119 (921h3m23.967860227s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615308858 (921h3m23.968093966s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615443009 (921h3m23.968228107s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615976040 (921h3m23.968761148s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:12:18.505364225 (921h3m53.858149333s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505470433 (921h3m53.858255542s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505532189 (921h3m53.858340762s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: rpc: dial tcp 45.91.116.13:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505618622 (921h3m53.85840373s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:61408->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505657595 (921h3m53.858442703s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505699794 (921h3m53.858484892s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 5.161.89.252:38782->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505756741 (921h3m53.858541849s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 5.161.89.252:28690->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505786897 (921h3m53.858572005s) [seg_pos={0 0}, upload=810581]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:12:18.523379844 (921h3m53.876164942s) [upload=810581]: Done uploading segments...
Oct 6 23:12:18.523393139 (921h3m53.876178237s) [upload=810581]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 1.506115357s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
702.191848ms
FAIL (2025-10-06 23:12:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 702.207097ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.015218802s
FAIL (2025-10-06 23:12:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
:35.809563147 (921h3m11.162348244s) [seg_pos={0 0}, upload=810577]: Begin upload segment...
Oct 6 23:11:35.809587702 (921h3m11.1623728s) [seg_pos={0 0}, upload=810577]: Done begin upload segment.
Oct 6 23:11:35.809607890 (921h3m11.162392988s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809621256 (921h3m11.162406353s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809862639 (921h3m11.162647737s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.809935506 (921h3m11.162720614s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810072883 (921h3m11.162857982s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810201284 (921h3m11.162986382s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810333482 (921h3m11.16311859s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810503220 (921h3m11.163288328s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810550219 (921h3m11.163335317s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810752629 (921h3m11.163537737s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810765003 (921h3m11.163550111s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.810957553 (921h3m11.163742652s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811023277 (921h3m11.163808385s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811167938 (921h3m11.163953036s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811293383 (921h3m11.164078491s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811417035 (921h3m11.164202134s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811637500 (921h3m11.164422608s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.811840040 (921h3m11.164625138s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Uploading piece...
Oct 6 23:11:35.876127968 (921h3m11.228913076s) [upload=810577]: Next returned nil segment
Oct 6 23:11:35.876140171 (921h3m11.228925269s) [upload=810577]: Waiting for error group managing segments...
Oct 6 23:11:35.883432457 (921h3m11.236217565s) [noise=true , address=145.79.227.39:4034 , piece_id=O23VFHC7FMIZS3QAR2IGDTMOPH76CRDIVUBNAKYW6WYTGVFUU6YA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883754090 (921h3m11.236539198s) [noise=true , address=145.79.227.43:4180 , piece_id=46G7IFIZYMPA7O23LLXUPVMF4Z2Z355GRVOBQSXTMBEEVF47PHKA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.883964114 (921h3m11.236749212s) [noise=true , address=145.79.227.25:4638 , piece_id=HWMQUW7VPAPTYF7VNIZTKQMXTFAS5QHU5LEVVMGGSBT33L4QBHSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885045895 (921h3m11.237831003s) [noise=true , address=145.79.227.22:4542 , piece_id=GQJC3V5EVADLPSIIZX3WLFMDF4MGSOLIBOUPNLKMK6M5S6TU32ZQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885329446 (921h3m11.238114545s) [noise=true , address=145.79.227.27:4710 , piece_id=MGHHT2XK7KWLVD7VN7Z66MMOJNRWHDNDS7BC7BWEPGJWANSUTMUA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885434994 (921h3m11.238220103s) [noise=true , address=145.79.227.52:4467 , piece_id=72BPXV7MQ6HL6IYVRJY5ZYNZXBZK5DANY7DYUDHZN3JOXMMR335A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.885700733 (921h3m11.238485831s) [noise=true , address=145.79.227.35:4487 , piece_id=IZ6PDJZ3PBYENUXSLIRF77B26GY3HR7PJNV2G5YSYBCMU4QFUM6A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:35.886007770 (921h3m11.238792878s) [noise=true , address=145.79.227.46:4636 , piece_id=M3X5SGNDJ2W2DFXMHYRCZC2U6KP65WFOPAXOE4OV2ZJXQ3JZ6U5Q, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.386975923 (921h3m11.739761052s) [noise=true , address=64.49.13.170:4728 , piece_id=YVQWH6JAW5H2PPDRBPOUEJEP4JEXJMJOFI6ZRA7QBTOY4FRVUUWA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.400346556 (921h3m11.753131684s) [noise=true , address=64.49.13.138:4483 , piece_id=RO2BRPFRTTOEKRSK46U57UOOU23OID2MHATKKER22REAW6LZ77OQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:11:36.430989544 (921h3m11.783774671s) [noise=true , address=64.49.13.26:4954 , piece_id=5ALOFIE25F4GXUMOOMCDUGHLC3VJWOGGZY4GGHS4KPYZVPVL5PRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:32.124753979 (921h4m7.477539087s) [noise=true , address=45.91.116.4:4969 , piece_id=OPFYPTUJ6GVURZ4VI6TOWJ4ZTVNZRVRNYXU7EILEXRDRTNEPZA2A, seg_pos={0 0}, upload=810577]: Done uploading piece. err: <nil>
Oct 6 23:12:35.776881675 (921h4m11.129666802s) [noise=true , address=77.90.31.15:4717 , piece_id=VKI6VCH2NSWK5ILM7UMVAA4GYXM6VAVHYFLH4ZRYUKS6MLVMOXXA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 5.161.89.252:21828->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776907392 (921h4m11.129692521s) [noise=true , address=45.91.116.10:4086 , piece_id=N7NDOOXZBKQESXBSOYOS6PMJ2ZPNVKX2UPTCGFQRMIJXGGGYARSQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 5.161.89.252:46224->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.776967846 (921h4m11.129752964s) [noise=true , address=77.90.31.10:4543 , piece_id=YSSNJK5GCFP6DRT52L3IFSAEKCXUYEI4JVEPTMZCKP5HZ2SSHNRQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: read tcp 5.161.89.252:36754->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777013391 (921h4m11.12979852s) [noise=true , address=77.90.31.13:4629 , piece_id=MEFBFMFDLYEDAUCX5Z57CZPPXF23PJ7SPOOS6VBWRDARZZR7WYNQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 5.161.89.252:50000->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777031185 (921h4m11.129816313s) [noise=true , address=77.90.31.17:4793 , piece_id=2RHB45CNKCHCJBO2JFXDUW7LTRHB4ZXVO4AU2HK2ZJMHTDD7LWOQ, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: write tcp 5.161.89.252:34804->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777073133 (921h4m11.129858262s) [noise=true , address=45.91.116.8:4058 , piece_id=UCOVQWF2LMYVXOSJWSTBYUWBWPQWKGDB5BGANPJAP5FCFIQMTUDA, seg_pos={0 0}, upload=810577]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 5.161.89.252:36286->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:35.777092450 (921h4m11.129877578s) [seg_pos={0 0}, upload=810577]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:12:35.790072881 (921h4m11.142858008s) [upload=810577]: Done uploading segments...
Oct 6 23:12:35.790116883 (921h4m11.142902011s) [upload=810577]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.015236115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X......X.] duration:
28.225600005s
FAIL (2025-10-06 23:12:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.225614192s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................X..........XX..........X.....X] duration:
30.020161145s
FAIL (2025-10-06 23:12:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:11:48.517381839 (921h3m23.870166937s) [upload=810581]: Starting upload
Oct 6 23:11:48.517430841 (921h3m23.870215949s) [upload=810581]: Done starting upload
Oct 6 23:11:48.517516051 (921h3m23.870301149s) [upload=810581]: Uploading segments...
Oct 6 23:11:48.517644803 (921h3m23.8704299s) [upload=810581]: Got next segment. Inline: false
Oct 6 23:11:48.517676692 (921h3m23.8704618s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:11:48.517680208 (921h3m23.870465317s) [upload=810581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:11:48.538737567 (921h3m23.891522675s) [seg_pos={0 0}, upload=810581]: Begin upload segment...
Oct 6 23:11:48.538764999 (921h3m23.891550107s) [seg_pos={0 0}, upload=810581]: Done begin upload segment.
Oct 6 23:11:48.538783212 (921h3m23.89156832s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.538801406 (921h3m23.891586505s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539074769 (921h3m23.891859867s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539078877 (921h3m23.891863985s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539310673 (921h3m23.892095771s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539328626 (921h3m23.892113725s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539541786 (921h3m23.892326885s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539546105 (921h3m23.892331202s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539757621 (921h3m23.892542729s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.539765105 (921h3m23.892550203s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540007210 (921h3m23.892792318s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540070559 (921h3m23.892855667s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540246559 (921h3m23.893031657s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.540660275 (921h3m23.893445373s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.541324632 (921h3m23.89410974s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.542295153 (921h3m23.895080261s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Uploading piece...
Oct 6 23:11:48.603736951 (921h3m23.956522059s) [upload=810581]: Next returned nil segment
Oct 6 23:11:48.603748833 (921h3m23.956533941s) [upload=810581]: Waiting for error group managing segments...
Oct 6 23:11:48.612397173 (921h3m23.965182271s) [noise=true , address=145.79.227.14:4838 , piece_id=PVJCNLF423EGC3U4YUJVMWSW5YZKU2GNFU35SL6NBERJHYMHMQVQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.612520434 (921h3m23.965305542s) [noise=true , address=145.79.227.6:4257 , piece_id=N75SNWR7CIPS3IU7DXVZ4LGXP5HS6LJMCY77DM4QJTIM3OLNCSUA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613048776 (921h3m23.965833874s) [noise=true , address=145.79.227.13:4801 , piece_id=RHZHIVA7Y3V2VHYZIW5D66E5ESHJHTFFIQ7DOYU7MX7MIJJ2XGDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.613406328 (921h3m23.966191426s) [noise=true , address=145.79.227.4:4185 , piece_id=FZZQVGXBZFZUTFAUU4PFK44Y3MISMWLRC2DMI3NF7T2ZL6ZNDSHA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.614573888 (921h3m23.967358987s) [noise=true , address=145.79.227.18:4007 , piece_id=RRW6L5KIKRPI5LZ75VXL2R37VFRVGDC4NEXJCLMTXIGY6E7TQSMQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615075119 (921h3m23.967860227s) [noise=true , address=145.79.227.15:4873 , piece_id=4GP5KIELQKJL5CRMBPQK5YEH65WP34RILTRP4PL732LEFR443GOQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615308858 (921h3m23.968093966s) [noise=true , address=145.79.227.13:4804 , piece_id=RNEMLB6PFUHPJUVH55CC5GADMBUQY5VL63HCUJNE5ZE7CFMRJJVA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615443009 (921h3m23.968228107s) [noise=true , address=145.79.227.11:4730 , piece_id=VPAJALXPHOKQ7ILAF6NEIOFTOLEWR6SV7YG3SB6J247QQFXKZVBA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:11:48.615976040 (921h3m23.968761148s) [noise=true , address=145.79.227.3:4150 , piece_id=XRSW72YCOHEZSSH7F57OVIGSCKKGOPM4DCTONKWFDCN5ETUCGEDA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: <nil>
Oct 6 23:12:18.505364225 (921h3m53.858149333s) [noise=true , address=77.90.31.17:4771 , piece_id=TZZO76XW3CP6XZUIEOQXT6SZXUZFIQM3BYJNVNJJH75MJIZAPBIQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505470433 (921h3m53.858255542s) [noise=true , address=45.91.116.7:4009 , piece_id=BNJIAFPZ7DVSJAOLSK7J6HPEIN5KHV7KTWU7GE5AD225KWMYTZYQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505532189 (921h3m53.858340762s) [noise=true , address=45.91.116.13:4154 , piece_id=UQKIEU3DKKGRGSLLAZSDCXYZR7AN4JKY43WUZXJCU7REGGRNS4JQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: rpc: dial tcp 45.91.116.13:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505618622 (921h3m53.85840373s) [noise=true , address=45.91.116.12:4227 , piece_id=YQZVMKGQ336TMDH2UFRZPMV7RKIHZ2CGH3P7UXH5IKPTID7UIXFQ, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:61408->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505657595 (921h3m53.858442703s) [noise=true , address=77.90.31.21:4877 , piece_id=QAHDR74H7DQHES3ANPHMB6YLBIC3QB57GEDEZC7VW6DSJTQGCUFA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505699794 (921h3m53.858484892s) [noise=true , address=45.91.116.11:4118 , piece_id=ISFUQJJCPEDAZLUG3KBMH2237L4PSJXJGMR2G57JZNWBBLUKLETA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 5.161.89.252:38782->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505756741 (921h3m53.858541849s) [noise=true , address=77.90.31.10:4517 , piece_id=EZFGXC42CRSNPYEX4WNOBXCC5GXR4DZ3SUGLYBDRPSROWBJMMFLA, seg_pos={0 0}, upload=810581]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 5.161.89.252:28690->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:18.505786897 (921h3m53.858572005s) [seg_pos={0 0}, upload=810581]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:12:18.523379844 (921h3m53.876164942s) [upload=810581]: Done uploading segments...
Oct 6 23:12:18.523393139 (921h3m53.876178237s) [upload=810581]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.020177837s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X......X] duration:
30.001197938s
FAIL (2025-10-06 23:12:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001215049s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X..........XX..........X.....] duration:
672.059231ms
FAIL (2025-10-06 23:11:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 672.085529ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
615.7585ms
FAIL (2025-10-06 23:11:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 615.780962ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X..........XX..........X....] duration:
379.809405ms
FAIL (2025-10-06 23:11:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 379.830434ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X..........XX..........X...] duration:
504.097647ms
FAIL (2025-10-06 23:10:41 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 504.118115ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
516.282864ms
FAIL (2025-10-06 23:10:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 516.299406ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
733.67445ms
FAIL (2025-10-06 23:10:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 733.691401ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
953.442139ms
FAIL (2025-10-06 23:09:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 953.459161ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................X..........XX..........X..] duration:
578.22994ms
FAIL (2025-10-06 23:09:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 578.244478ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
10.700722356s
FAIL (2025-10-06 23:09:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.700736633s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X..........XX..........X.] duration:
29.109010591s
FAIL (2025-10-06 23:09:37 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 29.10902574s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................X..........XX..........X] duration:
30.019132455s
FAIL (2025-10-06 23:09:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:08:38.622381101 (921h0m13.975166199s) [upload=810531]: Starting upload
Oct 6 23:08:38.622433970 (921h0m13.975219078s) [upload=810531]: Done starting upload
Oct 6 23:08:38.622535542 (921h0m13.97532065s) [upload=810531]: Uploading segments...
Oct 6 23:08:38.622670365 (921h0m13.975455473s) [upload=810531]: Got next segment. Inline: false
Oct 6 23:08:38.622700442 (921h0m13.975485539s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:08:38.622704118 (921h0m13.975489216s) [upload=810531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:08:38.645160252 (921h0m13.99794536s) [seg_pos={0 0}, upload=810531]: Begin upload segment...
Oct 6 23:08:38.645189176 (921h0m13.997974274s) [seg_pos={0 0}, upload=810531]: Done begin upload segment.
Oct 6 23:08:38.645207340 (921h0m13.997992448s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645216097 (921h0m13.998001194s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645498356 (921h0m13.998283454s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645606679 (921h0m13.998391787s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645729500 (921h0m13.998514597s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645925367 (921h0m13.998710465s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.645950594 (921h0m13.998735702s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646156542 (921h0m13.99894165s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646218708 (921h0m13.999003816s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646397894 (921h0m13.999182992s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646564917 (921h0m13.999350025s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646793456 (921h0m13.999578564s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.646958957 (921h0m13.999744065s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647134988 (921h0m13.999920096s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647228242 (921h0m14.000013351s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.647368345 (921h0m14.000153443s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Uploading piece...
Oct 6 23:08:38.742608181 (921h0m14.095393309s) [upload=810531]: Next returned nil segment
Oct 6 23:08:38.742651242 (921h0m14.09543637s) [upload=810531]: Waiting for error group managing segments...
Oct 6 23:08:38.749330257 (921h0m14.102115374s) [noise=true , address=145.79.227.4:4187 , piece_id=NG2ZUYHO6JHB4Y42G4IU5KRTSSIT574QSNK7YM2S2L2ECQMESDLA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.750573330 (921h0m14.103358438s) [noise=true , address=145.79.227.3:4150 , piece_id=U2XEYJ3BZEURMUFYWPRPLX3OR6Z6MSOLBZTVV4YBHVF5RIIJLTQQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.751449464 (921h0m14.104234592s) [noise=true , address=145.79.227.11:4730 , piece_id=5RBBG5OUBG2JHETW6NH4GBBG42IYGAXP6QN52F47XQQMJZDE56EQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752165909 (921h0m14.104951027s) [noise=true , address=145.79.227.4:4186 , piece_id=SN6CHFOJ5QCDAWIJHSJQAC2HMWKLMUO3MEE5K6USJCUE3HEJJLSA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.752787484 (921h0m14.105572613s) [noise=true , address=145.79.227.17:4971 , piece_id=SWQRXDQXWQ5HTG4ADTZNRDKH2QRY6XJA6GSRR7QYXTBUTDGMLT3A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.753562729 (921h0m14.106347858s) [noise=true , address=145.79.227.2:4115 , piece_id=Q6GEJ6232PNKK6EYW7V56U5UF4QXBFC5RMQ62AHN5XJODJEC3KHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754240872 (921h0m14.107026s) [noise=true , address=145.79.227.7:4295 , piece_id=NHAT2GMKBO5GPSP3U3OTZKEXNY56AULL4ZZEUFTUNK472KFXYQCA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.754842732 (921h0m14.107627861s) [noise=true , address=145.79.227.16:4909 , piece_id=UY5W72A4AAGAC2WVR3FBGU7PMUEBJIQDWUVF2NBXVPTAG4A62B6A, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.755505116 (921h0m14.108290243s) [noise=true , address=145.79.227.6:4259 , piece_id=JEWNU42SS3BWDV4TGIS4PN4O7ZFWTB3AQBTTLCXLOCA5BCAXXETA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756498780 (921h0m14.109283898s) [noise=true , address=145.79.227.15:4876 , piece_id=JZCVIIT3NGQBN2FKPO4GKAJEZITVELQRVZJHNDK3C6RIG357VDSQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:08:38.756708874 (921h0m14.109494003s) [noise=true , address=145.79.227.5:4222 , piece_id=BILCFQT42QYDRFK4ZHUNPI3KPK7MZEFPK74M4RXAAGEHGMLNKNMA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: <nil>
Oct 6 23:09:08.595773978 (921h0m43.948559107s) [noise=true , address=45.91.116.12:4226 , piece_id=MSWXEYW2OXEZFWVKJ3WD2SLFUJFJXDDI4OCMMTHREQ5BT3YGOGHA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595841165 (921h0m43.948626282s) [noise=true , address=45.91.116.5:4298 , piece_id=PDWIIQ3LKZ4S2C4PBCFPJINRKTQJFEZ7ZB5DA7YLCS2TZDMBXMRQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 5.161.89.252:40386->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595853818 (921h0m43.948638936s) [noise=true , address=77.90.31.16:4734 , piece_id=IT4VFPE67QTW5JOQRW67SCO4UARF4IA6REVDK53HFHVYAA5WPWBA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:13860->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595877402 (921h0m43.94866253s) [noise=true , address=77.90.31.21:4877 , piece_id=6E6MR4BFYLOWNMQK7VIUIJ2B6U73B4Z6EQGHBPSRKMY2K5YMQGHQ, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595947223 (921h0m43.948732352s) [noise=true , address=45.91.116.10:4081 , piece_id=UUTQD67GSZZPRVN4DWRWYX7QRYUMJSUGVLZ3LXW2VCP7AZVKJXWA, seg_pos={0 0}, upload=810531]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:57036->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:08.595978832 (921h0m43.94876396s) [seg_pos={0 0}, upload=810531]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:09:08.612159078 (921h0m43.964944216s) [upload=810531]: Done uploading segments...
Oct 6 23:09:08.612179366 (921h0m43.964964494s) [upload=810531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.019165357s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
472.204934ms
FAIL (2025-10-06 23:08:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.226745ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
873.252387ms
FAIL (2025-10-06 23:08:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 873.3132ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
687.646552ms
FAIL (2025-10-06 23:08:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 687.657052ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
587.835236ms
FAIL (2025-10-06 23:07:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.855423ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
592.660352ms
FAIL (2025-10-06 23:07:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 592.676952ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X..........XX..........] duration:
740.104809ms
FAIL (2025-10-06 23:07:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 22:59:46.884077630 (920h51m22.236862799s) [upload=810403]: Starting upload
Oct 6 22:59:46.884205150 (920h51m22.236990278s) [upload=810403]: Done starting upload
Oct 6 22:59:46.884295620 (920h51m22.237080757s) [upload=810403]: Uploading segments...
Oct 6 22:59:46.884666616 (920h51m22.237451745s) [upload=810403]: Got next segment. Inline: false
Oct 6 22:59:46.884743290 (920h51m22.237528418s) [upload=810403]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:46.884750263 (920h51m22.237535391s) [upload=810403]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:46.904611888 (920h51m22.257397016s) [seg_pos={0 0}, upload=810403]: Begin upload segment...
Oct 6 22:59:46.904679865 (920h51m22.257464993s) [seg_pos={0 0}, upload=810403]: Done begin upload segment.
Oct 6 22:59:46.904733516 (920h51m22.257518644s) [noise=true , address=45.91.116.13:4155 , piece_id=XMU42CCMKBPTP6M2J63OVXS6H4K3NAI3MO34DH7JAYJ5PWMZXA5A, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.904830868 (920h51m22.257615997s) [noise=true , address=145.79.227.4:4185 , piece_id=2E3UKKXOXO3T4JIEDDWJQZCKQPYAO3SJJI6QVO5PYIY4FOV5C2QA, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.905315337 (920h51m22.258100455s) [noise=true , address=145.79.227.17:4970 , piece_id=73Q54SKV7OWC6IPYPSBGVVL4CWIMIX6U3NPVRK43OW2Q452PJXYQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.905607857 (920h51m22.258392994s) [noise=true , address=145.79.227.11:4729 , piece_id=3WZERYWGJKFIL5G4YU4H6RKIQAORL4M7VPSVIDV64OCZOTB42X4Q, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.905957051 (920h51m22.25874218s) [noise=true , address=145.79.227.12:4767 , piece_id=434JR5VGSKXMFPPVGBOP5Q3L5ROEBLRXOZW6LGURBEEYPS4GTUGQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.906281602 (920h51m22.259066729s) [noise=true , address=77.90.31.16:4735 , piece_id=IY36QFNJOR56TF5ZJ7R4WW5FUTAQJQBBDMDH7N7NFTCE44TSPIHQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.906518225 (920h51m22.259303353s) [noise=true , address=77.90.31.17:4771 , piece_id=EOAZY4XGVWL2T7BSXKUBLNYFYY2RX4DZRQVMONIGHPQD7DXAIXUQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.906917053 (920h51m22.259702181s) [noise=true , address=145.79.227.15:4873 , piece_id=M7AYCJ6RZHCSAFDCQ7Q5FOY6JT3CTCJX3KDVHB2S37EGO2LYWMOQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.907043660 (920h51m22.259828789s) [noise=true , address=145.79.227.14:4839 , piece_id=DC2C7XPYH2NXHPPWSJAA5NN3X4JFLEKQCRTO4KT4PTQBM6GOQRWQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.907597981 (920h51m22.260383109s) [noise=true , address=77.90.31.14:4663 , piece_id=QDFRXC5PBKH2TP6XTJMNYTCZDNKR35OYPSRKRIIA3QBPDQXEMCBQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.907654597 (920h51m22.260439736s) [noise=true , address=45.91.116.13:4154 , piece_id=K4CLNNA2UMBAD5EF3LD5UYDN7YRJBGK5WXJPF7BQMTB6SR7LWRQA, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.908134929 (920h51m22.260920057s) [noise=true , address=145.79.227.18:4006 , piece_id=CJJK5TQBA3K7I47PONSL3CJOWV7XVHJFSKO4SZZSAPVL546O3YYA, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.908312362 (920h51m22.2610975s) [noise=true , address=145.79.227.11:4732 , piece_id=VXV7ZIPUBIMWUPGOPV47KCNAG6HMLWUSK3OK3KB6ZKHNEDNINYUQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.908796781 (920h51m22.26158191s) [noise=true , address=145.79.227.14:4840 , piece_id=XJV5WADZHNE7FYURZSPNRCSQ7XIEPDSLONLWC3MRIVG4DN5QACCQ, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.911190102 (920h51m22.26397523s) [noise=true , address=145.79.227.13:4804 , piece_id=CV2BI7EVHLI36A26ZFUOHBKCG32MFRLE5E75I5QFY22HDK54NS4A, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:46.912062269 (920h51m22.264847397s) [noise=true , address=145.79.227.18:4005 , piece_id=NZEFMJLR54B7PLIKOABGWBD7LLUW77QUQOQACAO4CH7IGUL7FHIA, seg_pos={0 0}, upload=810403]: Uploading piece...
Oct 6 22:59:47.008827498 (920h51m22.361612616s) [upload=810403]: Next returned nil segment
Oct 6 22:59:47.008841855 (920h51m22.361626963s) [upload=810403]: Waiting for error group managing segments...
Oct 6 22:59:47.016799659 (920h51m22.369584767s) [noise=true , address=145.79.227.11:4732 , piece_id=VXV7ZIPUBIMWUPGOPV47KCNAG6HMLWUSK3OK3KB6ZKHNEDNINYUQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.017144386 (920h51m22.369929494s) [noise=true , address=145.79.227.15:4873 , piece_id=M7AYCJ6RZHCSAFDCQ7Q5FOY6JT3CTCJX3KDVHB2S37EGO2LYWMOQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.017266185 (920h51m22.370051283s) [noise=true , address=145.79.227.14:4839 , piece_id=DC2C7XPYH2NXHPPWSJAA5NN3X4JFLEKQCRTO4KT4PTQBM6GOQRWQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.017878554 (920h51m22.370663652s) [noise=true , address=145.79.227.11:4729 , piece_id=3WZERYWGJKFIL5G4YU4H6RKIQAORL4M7VPSVIDV64OCZOTB42X4Q, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.018255210 (920h51m22.371040339s) [noise=true , address=145.79.227.17:4970 , piece_id=73Q54SKV7OWC6IPYPSBGVVL4CWIMIX6U3NPVRK43OW2Q452PJXYQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.018400854 (920h51m22.371185962s) [noise=true , address=145.79.227.18:4006 , piece_id=CJJK5TQBA3K7I47PONSL3CJOWV7XVHJFSKO4SZZSAPVL546O3YYA, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.018943813 (920h51m22.371728921s) [noise=true , address=145.79.227.14:4840 , piece_id=XJV5WADZHNE7FYURZSPNRCSQ7XIEPDSLONLWC3MRIVG4DN5QACCQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.019253594 (920h51m22.372038682s) [noise=true , address=145.79.227.4:4185 , piece_id=2E3UKKXOXO3T4JIEDDWJQZCKQPYAO3SJJI6QVO5PYIY4FOV5C2QA, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.019700543 (920h51m22.372506441s) [noise=true , address=145.79.227.12:4767 , piece_id=434JR5VGSKXMFPPVGBOP5Q3L5ROEBLRXOZW6LGURBEEYPS4GTUGQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.020258390 (920h51m22.373043498s) [noise=true , address=145.79.227.13:4804 , piece_id=CV2BI7EVHLI36A26ZFUOHBKCG32MFRLE5E75I5QFY22HDK54NS4A, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 22:59:47.028448570 (920h51m22.381233678s) [noise=true , address=145.79.227.18:4005 , piece_id=NZEFMJLR54B7PLIKOABGWBD7LLUW77QUQOQACAO4CH7IGUL7FHIA, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 23:00:16.612044719 (920h51m51.964829848s) [noise=true , address=45.91.116.13:4154 , piece_id=K4CLNNA2UMBAD5EF3LD5UYDN7YRJBGK5WXJPF7BQMTB6SR7LWRQA, seg_pos={0 0}, upload=810403]: Done uploading piece. err: <nil>
Oct 6 23:00:16.863245986 (920h51m52.216031115s) [noise=true , address=77.90.31.17:4771 , piece_id=EOAZY4XGVWL2T7BSXKUBLNYFYY2RX4DZRQVMONIGHPQD7DXAIXUQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 5.161.89.252:34364->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:16.863316969 (920h51m52.216102098s) [noise=true , address=45.91.116.13:4155 , piece_id=XMU42CCMKBPTP6M2J63OVXS6H4K3NAI3MO34DH7JAYJ5PWMZXA5A, seg_pos={0 0}, upload=810403]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 5.161.89.252:38022->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:16.863413090 (920h51m52.216198218s) [noise=true , address=77.90.31.16:4735 , piece_id=IY36QFNJOR56TF5ZJ7R4WW5FUTAQJQBBDMDH7N7NFTCE44TSPIHQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:16.863464186 (920h51m52.216249313s) [noise=true , address=77.90.31.14:4663 , piece_id=QDFRXC5PBKH2TP6XTJMNYTCZDNKR35OYPSRKRIIA3QBPDQXEMCBQ, seg_pos={0 0}, upload=810403]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:16.863511255 (920h51m52.216296383s) [seg_pos={0 0}, upload=810403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:00:16.887849070 (920h51m52.240634208s) [upload=810403]: Done uploading segments...
Oct 6 23:00:16.887877603 (920h51m52.240662731s) [upload=810403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 740.125137ms
core
uplink
us-select-1
|