Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X...X...X..] duration:
128.019557ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:39:22 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 128.072296ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000655902s
FAIL (2025-08-07 21:39:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000675908s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.0015374s
FAIL (2025-08-07 21:39:15 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5611523 (5h2m18.672667733s) [upload=5174]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:05.172798369 (5h2m18.75985459s) [upload=5191]: Starting upload Aug 7 21:29:05.172850326 (5h2m18.759906537s) [upload=5191]: Done starting upload Aug 7 21:29:05.172893617 (5h2m18.759949839s) [upload=5191]: Uploading segments... Aug 7 21:29:05.172921278 (5h2m18.759977499s) [upload=5191]: Got next segment. Inline: true Aug 7 21:29:05.172924825 (5h2m18.759981046s) [upload=5191]: Waiting for error group managing segments... Aug 7 21:29:05.172954671 (5h2m18.760010882s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:05.172958317 (5h2m18.760014538s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:05.172960632 (5h2m18.760016843s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:05.162199693 (5h3m18.749255944s) [upload=5191]: Done uploading segments... Aug 7 21:30:05.162220462 (5h3m18.749276703s) [upload=5191]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:05.259129610 (5h3m18.846185831s) [upload=5205]: Starting upload Aug 7 21:30:05.259178532 (5h3m18.846234742s) [upload=5205]: Done starting upload Aug 7 21:30:05.259225909 (5h3m18.846282131s) [upload=5205]: Uploading segments... Aug 7 21:30:05.259254723 (5h3m18.846310935s) [upload=5205]: Got next segment. Inline: true Aug 7 21:30:05.259258370 (5h3m18.846314591s) [upload=5205]: Waiting for error group managing segments... Aug 7 21:30:05.259289909 (5h3m18.846346131s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:05.259293515 (5h3m18.846349736s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:05.259295659 (5h3m18.84635187s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:05.245621000 (5h4m18.83267725s) [upload=5205]: Done uploading segments... Aug 7 21:31:05.245643692 (5h4m18.832699933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:15.177436931 (5h10m28.764493172s) [upload=5324]: Starting upload Aug 7 21:37:15.177533852 (5h10m28.764590103s) [upload=5324]: Done starting upload Aug 7 21:37:15.177602810 (5h10m28.764659061s) [upload=5324]: Uploading segments... Aug 7 21:37:15.177645340 (5h10m28.764701581s) [upload=5324]: Got next segment. Inline: true Aug 7 21:37:15.177650961 (5h10m28.764707202s) [upload=5324]: Waiting for error group managing segments... Aug 7 21:37:15.177698649 (5h10m28.76475489s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:15.177704300 (5h10m28.764760541s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:15.177708177 (5h10m28.764764428s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:40.730185715 (5h10m54.317241957s) [upload=5324]: Done uploading segments... Aug 7 21:37:40.730203639 (5h10m54.31725988s) [upload=5324]: Upload finished. err: <nil> Aug 7 21:38:15.287715124 (5h11m28.874771365s) [upload=5341]: Starting upload Aug 7 21:38:15.287780276 (5h11m28.874836516s) [upload=5341]: Done starting upload Aug 7 21:38:15.287844856 (5h11m28.874901107s) [upload=5341]: Uploading segments... Aug 7 21:38:15.287889961 (5h11m28.874946201s) [upload=5341]: Got next segment. Inline: true Aug 7 21:38:15.287895600 (5h11m28.874951842s) [upload=5341]: Waiting for error group managing segments... Aug 7 21:38:15.287984066 (5h11m28.875040317s) [upload=5341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:15.287990077 (5h11m28.875046318s) [upload=5341]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:15.287993564 (5h11m28.875049804s) [upload=5341]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:15.275679218 (5h12m28.862735459s) [upload=5341]: Done uploading segments... Aug 7 21:39:15.275708622 (5h12m28.862764883s) [upload=5341]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001561745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001267014s
FAIL (2025-08-07 21:39:07 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.00128675s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [..................................................X...X...X.] duration:
119.805815ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:39:05 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 119.821775ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.547751285s
FAIL (2025-08-07 21:39:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.547783513s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.00075239s
FAIL (2025-08-07 21:38:56 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:26.513648977 (5h11m40.100705218s) [upload=5344]: Starting upload Aug 7 21:38:26.513731310 (5h11m40.100787552s) [upload=5344]: Done starting upload Aug 7 21:38:26.513792245 (5h11m40.100848496s) [upload=5344]: Uploading segments... Aug 7 21:38:26.513834112 (5h11m40.100890354s) [upload=5344]: Got next segment. Inline: true Aug 7 21:38:26.513839653 (5h11m40.100895904s) [upload=5344]: Waiting for error group managing segments... Aug 7 21:38:26.513884006 (5h11m40.100940247s) [upload=5344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:26.513889947 (5h11m40.100946188s) [upload=5344]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:26.513893434 (5h11m40.100949675s) [upload=5344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:56.502052245 (5h12m10.089108537s) [upload=5344]: Done uploading segments... Aug 7 21:38:56.502102730 (5h12m10.089158972s) [upload=5344]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.00076824s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
35 out of 60 is failed (threshold=3) [.......................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.002256073s
FAIL (2025-08-07 21:38:51 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.00228139s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.207531684s
FAIL (2025-08-07 21:38:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.207557483s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [..................................XX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000553542s
FAIL (2025-08-07 21:38:39 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000570454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration:
1m0.000560046s
FAIL (2025-08-07 21:38:34 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
}, upload=5311]: Uploading piece...
Aug 7 21:37:03.825713783 (5h10m17.412770024s) [noise=true , address=64.49.13.155:4914 , piece_id=6AUGLHYNHGAKMNJRMNA74PJFKMCCCSKHK3ETQICHKCUGWFTWTCMQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826048337 (5h10m17.413104579s) [noise=true , address=104.36.82.23:4472 , piece_id=G45VA52TYEJYQU36RRE4H67F5JKFA63A6ZG67PU5KMCYTXAMZUUQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826090746 (5h10m17.413146987s) [noise=true , address=104.36.82.58:4287 , piece_id=ETEE347VQKLH2CZUDKNWEE3CDMSKQHMZ3NJGXUL7FL54IYPDFICA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826469293 (5h10m17.413525534s) [noise=true , address=104.36.82.53:4141 , piece_id=ED5MIJF6D2MOWFMGQ46XDSPTXVBEAYIBV3CSZEQ36LELLPJ6WZFQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826488740 (5h10m17.413544981s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826841057 (5h10m17.413897299s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826983933 (5h10m17.414040174s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827218521 (5h10m17.414274763s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827383179 (5h10m17.41443942s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827627445 (5h10m17.414683687s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827751647 (5h10m17.414807898s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828066434 (5h10m17.415122676s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828167473 (5h10m17.415223714s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828451093 (5h10m17.415507334s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828515743 (5h10m17.415571985s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.934711994 (5h10m17.521768225s) [upload=5311]: Next returned nil segment
Aug 7 21:37:03.934726611 (5h10m17.521782853s) [upload=5311]: Waiting for error group managing segments...
Aug 7 21:37:03.953545219 (5h10m17.540601461s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.958188677 (5h10m17.545244919s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.964982629 (5h10m17.552038869s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.969165096 (5h10m17.556221337s) [noise=true , address=104.36.82.19:4426 , piece_id=3QBNLWLB2C4ACZTZ276C4VHGPUYX7NWV3YWE3YJMSCCOUL442RHA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.973698649 (5h10m17.56075489s) [noise=true , address=104.36.82.53:4141 , piece_id=ED5MIJF6D2MOWFMGQ46XDSPTXVBEAYIBV3CSZEQ36LELLPJ6WZFQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.975101527 (5h10m17.562157768s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.975868810 (5h10m17.562925051s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.976647924 (5h10m17.563704165s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.981871894 (5h10m17.568928146s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.990165577 (5h10m17.577221818s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.996463181 (5h10m17.583519422s) [noise=true , address=77.90.31.16:4742 , piece_id=ERUGO4MG35TTSV76XZAL4DL76A5C3XQBGOT4SE3B3KYCNQLDGIXA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.067745290 (5h10m17.65480153s) [noise=true , address=104.36.82.47:4545 , piece_id=JI4XEDNFM2C3XUT5JD2Q5OFBN5EW77GVKKNB75BSZML24VIZNRNQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.194336421 (5h10m17.781392672s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.231214259 (5h10m17.8182705s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.231245878 (5h10m17.818302118s) [seg_pos={0 0}, upload=5311]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:37:04.231458835 (5h10m17.818515086s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: expected piece hash; context canceled; EOF
Aug 7 21:37:04.231571595 (5h10m17.818627836s) [noise=true , address=104.36.82.23:4472 , piece_id=G45VA52TYEJYQU36RRE4H67F5JKFA63A6ZG67PU5KMCYTXAMZUUQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: read tcp 5.161.89.252:51344->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:37:04.231596983 (5h10m17.818653224s) [noise=true , address=64.49.13.155:4914 , piece_id=6AUGLHYNHGAKMNJRMNA74PJFKMCCCSKHK3ETQICHKCUGWFTWTCMQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection
Aug 7 21:37:04.231638852 (5h10m17.818695092s) [noise=true , address=104.36.82.58:4287 , piece_id=ETEE347VQKLH2CZUDKNWEE3CDMSKQHMZ3NJGXUL7FL54IYPDFICA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wNKharWFFSSsLMd1vZYi6zYEYBe3VT6Z3ikrY7Cjo3xPLUK62): protocol: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection
Aug 7 21:37:04.231659821 (5h10m17.818716062s) [seg_pos={0 0}, upload=5311]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:37:04.231740732 (5h10m17.818796972s) [upload=5311]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:37:04.231746432 (5h10m17.818802674s) [upload=5311]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:37:34.607598692 (5h10m48.194654932s) [upload=5311]: Done uploading segments...
Aug 7 21:37:34.607648003 (5h10m48.194704254s) [upload=5311]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:37:34.704354810 (5h10m48.291411021s) [upload=5330]: Starting upload
Aug 7 21:37:34.704409994 (5h10m48.291466215s) [upload=5330]: Done starting upload
Aug 7 21:37:34.704465938 (5h10m48.291522179s) [upload=5330]: Uploading segments...
Aug 7 21:37:34.704648018 (5h10m48.291704229s) [upload=5330]: Got next segment. Inline: false
Aug 7 21:37:34.704676862 (5h10m48.291733083s) [upload=5330]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:37:34.704680678 (5h10m48.291736899s) [upload=5330]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:38:34.684727743 (5h11m48.271783975s) [upload=5330]: Next segment err: context deadline exceeded
Aug 7 21:38:34.684789619 (5h11m48.27184584s) [upload=5330]: Done uploading segments...
Aug 7 21:38:34.684808654 (5h11m48.271864865s) [upload=5330]: Upload finished. err: context deadline exceeded
Duration: 1m0.000570684s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001393432s
FAIL (2025-08-07 21:38:26 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h5m2.203431078s) [upload=5229]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:23.393738541 (5h5m36.980794762s) [upload=5242]: Starting upload Aug 7 21:32:23.393794196 (5h5m36.980850406s) [upload=5242]: Done starting upload Aug 7 21:32:23.393846161 (5h5m36.980902382s) [upload=5242]: Uploading segments... Aug 7 21:32:23.393884694 (5h5m36.980940915s) [upload=5242]: Got next segment. Inline: true Aug 7 21:32:23.393888612 (5h5m36.980944822s) [upload=5242]: Waiting for error group managing segments... Aug 7 21:32:23.393921854 (5h5m36.980978075s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:23.393925500 (5h5m36.980981721s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:23.393927785 (5h5m36.980984006s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.381802738 (5h6m6.968858979s) [upload=5242]: Done uploading segments... Aug 7 21:32:53.381824659 (5h6m6.9688809s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001437675s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001519004s
FAIL (2025-08-07 21:38:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001543911s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001530028s
FAIL (2025-08-07 21:38:25 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ine: false Aug 7 21:30:07.260984843 (5h3m20.848041084s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:07.260994751 (5h3m20.848050993s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:30:37.232137150 (5h3m50.81919336s) [upload=5206]: Next segment err: context deadline exceeded Aug 7 21:30:37.232307578 (5h3m50.819363789s) [upload=5206]: Done uploading segments... Aug 7 21:30:37.232355838 (5h3m50.819412059s) [upload=5206]: Upload finished. err: context deadline exceeded Aug 7 21:30:41.838696318 (5h3m55.425752559s) [upload=5217]: Starting upload Aug 7 21:30:41.838781527 (5h3m55.425837767s) [upload=5217]: Done starting upload Aug 7 21:30:41.838903785 (5h3m55.425960006s) [upload=5217]: Uploading segments... Aug 7 21:30:41.839179650 (5h3m55.426235901s) [upload=5217]: Got next segment. Inline: false Aug 7 21:30:41.839223772 (5h3m55.426280023s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:41.839229273 (5h3m55.426285513s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:11.826385247 (5h4m25.413441468s) [upload=5217]: Next segment err: context deadline exceeded Aug 7 21:31:11.826544515 (5h4m25.413600736s) [upload=5217]: Done uploading segments... Aug 7 21:31:11.826568178 (5h4m25.413624399s) [upload=5217]: Upload finished. err: context deadline exceeded Aug 7 21:31:11.924007243 (5h4m25.511063484s) [upload=5227]: Starting upload Aug 7 21:31:11.924116868 (5h4m25.511173108s) [upload=5227]: Done starting upload Aug 7 21:31:11.924293237 (5h4m25.511349458s) [upload=5227]: Uploading segments... Aug 7 21:31:11.924515282 (5h4m25.511571553s) [upload=5227]: Got next segment. Inline: false Aug 7 21:31:11.924580593 (5h4m25.511636844s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.924591073 (5h4m25.511647324s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:41.901407105 (5h4m55.488463316s) [upload=5227]: Next segment err: context deadline exceeded Aug 7 21:31:41.901545804 (5h4m55.488602015s) [upload=5227]: Done uploading segments... Aug 7 21:31:41.901568547 (5h4m55.488624747s) [upload=5227]: Upload finished. err: context deadline exceeded Aug 7 21:32:25.036046257 (5h5m38.623102498s) [upload=5243]: Starting upload Aug 7 21:32:25.036157154 (5h5m38.623213395s) [upload=5243]: Done starting upload Aug 7 21:32:25.036227865 (5h5m38.623284077s) [upload=5243]: Uploading segments... Aug 7 21:32:25.036538426 (5h5m38.623594667s) [upload=5243]: Got next segment. Inline: false Aug 7 21:32:25.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded
Duration: 30.001553693s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001318501s
FAIL (2025-08-07 21:38:17 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.001333971s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000480393s
FAIL (2025-08-07 21:38:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000492356s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001020037s
FAIL (2025-08-07 21:38:15 UTC)
operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:28:05.013391970 (5h1m18.600448181s) [upload=5157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:05.107129855 (5h1m18.694186067s) [upload=5174]: Starting upload Aug 7 21:28:05.107184377 (5h1m18.694240598s) [upload=5174]: Done starting upload Aug 7 21:28:05.107240842 (5h1m18.694297063s) [upload=5174]: Uploading segments... Aug 7 21:28:05.107277301 (5h1m18.694333512s) [upload=5174]: Got next segment. Inline: true Aug 7 21:28:05.107280647 (5h1m18.694336868s) [upload=5174]: Waiting for error group managing segments... Aug 7 21:28:05.107312757 (5h1m18.694368969s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:05.107316373 (5h1m18.694372584s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:05.107318867 (5h1m18.694375088s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:05.085592777 (5h2m18.672648989s) [upload=5174]: Done uploading segments... Aug 7 21:29:05.085611523 (5h2m18.672667733s) [upload=5174]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:05.172798369 (5h2m18.75985459s) [upload=5191]: Starting upload Aug 7 21:29:05.172850326 (5h2m18.759906537s) [upload=5191]: Done starting upload Aug 7 21:29:05.172893617 (5h2m18.759949839s) [upload=5191]: Uploading segments... Aug 7 21:29:05.172921278 (5h2m18.759977499s) [upload=5191]: Got next segment. Inline: true Aug 7 21:29:05.172924825 (5h2m18.759981046s) [upload=5191]: Waiting for error group managing segments... Aug 7 21:29:05.172954671 (5h2m18.760010882s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:05.172958317 (5h2m18.760014538s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:05.172960632 (5h2m18.760016843s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:05.162199693 (5h3m18.749255944s) [upload=5191]: Done uploading segments... Aug 7 21:30:05.162220462 (5h3m18.749276703s) [upload=5191]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:05.259129610 (5h3m18.846185831s) [upload=5205]: Starting upload Aug 7 21:30:05.259178532 (5h3m18.846234742s) [upload=5205]: Done starting upload Aug 7 21:30:05.259225909 (5h3m18.846282131s) [upload=5205]: Uploading segments... Aug 7 21:30:05.259254723 (5h3m18.846310935s) [upload=5205]: Got next segment. Inline: true Aug 7 21:30:05.259258370 (5h3m18.846314591s) [upload=5205]: Waiting for error group managing segments... Aug 7 21:30:05.259289909 (5h3m18.846346131s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:05.259293515 (5h3m18.846349736s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:05.259295659 (5h3m18.84635187s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:05.245621000 (5h4m18.83267725s) [upload=5205]: Done uploading segments... Aug 7 21:31:05.245643692 (5h4m18.832699933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:15.177436931 (5h10m28.764493172s) [upload=5324]: Starting upload Aug 7 21:37:15.177533852 (5h10m28.764590103s) [upload=5324]: Done starting upload Aug 7 21:37:15.177602810 (5h10m28.764659061s) [upload=5324]: Uploading segments... Aug 7 21:37:15.177645340 (5h10m28.764701581s) [upload=5324]: Got next segment. Inline: true Aug 7 21:37:15.177650961 (5h10m28.764707202s) [upload=5324]: Waiting for error group managing segments... Aug 7 21:37:15.177698649 (5h10m28.76475489s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:15.177704300 (5h10m28.764760541s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:15.177708177 (5h10m28.764764428s) [upload=5324]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:40.730185715 (5h10m54.317241957s) [upload=5324]: Done uploading segments... Aug 7 21:37:40.730203639 (5h10m54.31725988s) [upload=5324]: Upload finished. err: <nil>
Duration: 1m0.00103815s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
34 out of 60 is failed (threshold=3) [........................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001361505s
FAIL (2025-08-07 21:38:05 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001379199s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000429059s
FAIL (2025-08-07 21:37:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000494192s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000926702s
FAIL (2025-08-07 21:37:54 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4145378 (5h4m30.721201618s) [upload=5219]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:18.629774505 (5h4m32.216830716s) [upload=5229]: Starting upload Aug 7 21:31:18.629831501 (5h4m32.216887712s) [upload=5229]: Done starting upload Aug 7 21:31:18.629882436 (5h4m32.216938658s) [upload=5229]: Uploading segments... Aug 7 21:31:18.629913235 (5h4m32.216969445s) [upload=5229]: Got next segment. Inline: true Aug 7 21:31:18.629916951 (5h4m32.216973162s) [upload=5229]: Waiting for error group managing segments... Aug 7 21:31:18.629950052 (5h4m32.217006263s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.629953899 (5h4m32.21701012s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:18.629956264 (5h4m32.217012475s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:48.616363445 (5h5m2.203419657s) [upload=5229]: Done uploading segments... Aug 7 21:31:48.616374867 (5h5m2.203431078s) [upload=5229]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:23.393738541 (5h5m36.980794762s) [upload=5242]: Starting upload Aug 7 21:32:23.393794196 (5h5m36.980850406s) [upload=5242]: Done starting upload Aug 7 21:32:23.393846161 (5h5m36.980902382s) [upload=5242]: Uploading segments... Aug 7 21:32:23.393884694 (5h5m36.980940915s) [upload=5242]: Got next segment. Inline: true Aug 7 21:32:23.393888612 (5h5m36.980944822s) [upload=5242]: Waiting for error group managing segments... Aug 7 21:32:23.393921854 (5h5m36.980978075s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:23.393925500 (5h5m36.980981721s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:23.393927785 (5h5m36.980984006s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.381802738 (5h6m6.968858979s) [upload=5242]: Done uploading segments... Aug 7 21:32:53.381824659 (5h6m6.9688809s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000946819s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001625114s
FAIL (2025-08-07 21:37:47 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.001643268s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [.....................................X........XX...X........] duration:
51.304417ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:37:47 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 51.330176ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.997688264s
FAIL (2025-08-07 21:37:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.997709504s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000748188s
FAIL (2025-08-07 21:37:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000763647s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [...................................XX.XXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001206767s
FAIL (2025-08-07 21:37:37 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001222075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
1m16.588794031s
FAIL (2025-08-07 21:37:34 UTC)
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2017 (5h9m31.623308258s) [upload=5311]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:37:03.825143368 (5h10m17.412199619s) [seg_pos={0 0}, upload=5311]: Begin upload segment...
Aug 7 21:37:03.825199522 (5h10m17.412255774s) [seg_pos={0 0}, upload=5311]: Done begin upload segment.
Aug 7 21:37:03.825228076 (5h10m17.412284317s) [noise=true , address=104.36.82.19:4426 , piece_id=3QBNLWLB2C4ACZTZ276C4VHGPUYX7NWV3YWE3YJMSCCOUL442RHA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.825258984 (5h10m17.412315225s) [noise=true , address=104.36.82.47:4545 , piece_id=JI4XEDNFM2C3XUT5JD2Q5OFBN5EW77GVKKNB75BSZML24VIZNRNQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.825689027 (5h10m17.412745268s) [noise=true , address=77.90.31.16:4742 , piece_id=ERUGO4MG35TTSV76XZAL4DL76A5C3XQBGOT4SE3B3KYCNQLDGIXA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.825713783 (5h10m17.412770024s) [noise=true , address=64.49.13.155:4914 , piece_id=6AUGLHYNHGAKMNJRMNA74PJFKMCCCSKHK3ETQICHKCUGWFTWTCMQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826048337 (5h10m17.413104579s) [noise=true , address=104.36.82.23:4472 , piece_id=G45VA52TYEJYQU36RRE4H67F5JKFA63A6ZG67PU5KMCYTXAMZUUQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826090746 (5h10m17.413146987s) [noise=true , address=104.36.82.58:4287 , piece_id=ETEE347VQKLH2CZUDKNWEE3CDMSKQHMZ3NJGXUL7FL54IYPDFICA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826469293 (5h10m17.413525534s) [noise=true , address=104.36.82.53:4141 , piece_id=ED5MIJF6D2MOWFMGQ46XDSPTXVBEAYIBV3CSZEQ36LELLPJ6WZFQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826488740 (5h10m17.413544981s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826841057 (5h10m17.413897299s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.826983933 (5h10m17.414040174s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827218521 (5h10m17.414274763s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827383179 (5h10m17.41443942s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827627445 (5h10m17.414683687s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.827751647 (5h10m17.414807898s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828066434 (5h10m17.415122676s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828167473 (5h10m17.415223714s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828451093 (5h10m17.415507334s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.828515743 (5h10m17.415571985s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Uploading piece...
Aug 7 21:37:03.934711994 (5h10m17.521768225s) [upload=5311]: Next returned nil segment
Aug 7 21:37:03.934726611 (5h10m17.521782853s) [upload=5311]: Waiting for error group managing segments...
Aug 7 21:37:03.953545219 (5h10m17.540601461s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.958188677 (5h10m17.545244919s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.964982629 (5h10m17.552038869s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.969165096 (5h10m17.556221337s) [noise=true , address=104.36.82.19:4426 , piece_id=3QBNLWLB2C4ACZTZ276C4VHGPUYX7NWV3YWE3YJMSCCOUL442RHA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.973698649 (5h10m17.56075489s) [noise=true , address=104.36.82.53:4141 , piece_id=ED5MIJF6D2MOWFMGQ46XDSPTXVBEAYIBV3CSZEQ36LELLPJ6WZFQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.975101527 (5h10m17.562157768s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.975868810 (5h10m17.562925051s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.976647924 (5h10m17.563704165s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.981871894 (5h10m17.568928146s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.990165577 (5h10m17.577221818s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:03.996463181 (5h10m17.583519422s) [noise=true , address=77.90.31.16:4742 , piece_id=ERUGO4MG35TTSV76XZAL4DL76A5C3XQBGOT4SE3B3KYCNQLDGIXA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.067745290 (5h10m17.65480153s) [noise=true , address=104.36.82.47:4545 , piece_id=JI4XEDNFM2C3XUT5JD2Q5OFBN5EW77GVKKNB75BSZML24VIZNRNQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.194336421 (5h10m17.781392672s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.231214259 (5h10m17.8182705s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil>
Aug 7 21:37:04.231245878 (5h10m17.818302118s) [seg_pos={0 0}, upload=5311]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:37:04.231458835 (5h10m17.818515086s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: expected piece hash; context canceled; EOF
Aug 7 21:37:04.231571595 (5h10m17.818627836s) [noise=true , address=104.36.82.23:4472 , piece_id=G45VA52TYEJYQU36RRE4H67F5JKFA63A6ZG67PU5KMCYTXAMZUUQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: read tcp 5.161.89.252:51344->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:37:04.231596983 (5h10m17.818653224s) [noise=true , address=64.49.13.155:4914 , piece_id=6AUGLHYNHGAKMNJRMNA74PJFKMCCCSKHK3ETQICHKCUGWFTWTCMQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection
Aug 7 21:37:04.231638852 (5h10m17.818695092s) [noise=true , address=104.36.82.58:4287 , piece_id=ETEE347VQKLH2CZUDKNWEE3CDMSKQHMZ3NJGXUL7FL54IYPDFICA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wNKharWFFSSsLMd1vZYi6zYEYBe3VT6Z3ikrY7Cjo3xPLUK62): protocol: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection
Aug 7 21:37:04.231659821 (5h10m17.818716062s) [seg_pos={0 0}, upload=5311]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:37:04.231740732 (5h10m17.818796972s) [upload=5311]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:37:04.231746432 (5h10m17.818802674s) [upload=5311]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:37:34.607598692 (5h10m48.194654932s) [upload=5311]: Done uploading segments...
Aug 7 21:37:34.607648003 (5h10m48.194704254s) [upload=5311]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m16.58881472s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001412061s
FAIL (2025-08-07 21:37:27 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== nt. Inline: false Aug 7 21:29:01.102903291 (5h2m14.689959502s) [upload=5189]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:01.102907839 (5h2m14.68996406s) [upload=5189]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:31.087349640 (5h2m44.67440585s) [upload=5189]: Next segment err: context deadline exceeded Aug 7 21:29:31.087429919 (5h2m44.67448613s) [upload=5189]: Done uploading segments... Aug 7 21:29:31.087444957 (5h2m44.674501178s) [upload=5189]: Upload finished. err: context deadline exceeded Aug 7 21:30:07.260405000 (5h3m20.847461241s) [upload=5206]: Starting upload Aug 7 21:30:07.260495098 (5h3m20.84755135s) [upload=5206]: Done starting upload Aug 7 21:30:07.260642223 (5h3m20.847698474s) [upload=5206]: Uploading segments... Aug 7 21:30:07.260927024 (5h3m20.847983276s) [upload=5206]: Got next segment. Inline: false Aug 7 21:30:07.260984843 (5h3m20.848041084s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:07.260994751 (5h3m20.848050993s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:30:37.232137150 (5h3m50.81919336s) [upload=5206]: Next segment err: context deadline exceeded Aug 7 21:30:37.232307578 (5h3m50.819363789s) [upload=5206]: Done uploading segments... Aug 7 21:30:37.232355838 (5h3m50.819412059s) [upload=5206]: Upload finished. err: context deadline exceeded Aug 7 21:30:41.838696318 (5h3m55.425752559s) [upload=5217]: Starting upload Aug 7 21:30:41.838781527 (5h3m55.425837767s) [upload=5217]: Done starting upload Aug 7 21:30:41.838903785 (5h3m55.425960006s) [upload=5217]: Uploading segments... Aug 7 21:30:41.839179650 (5h3m55.426235901s) [upload=5217]: Got next segment. Inline: false Aug 7 21:30:41.839223772 (5h3m55.426280023s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:41.839229273 (5h3m55.426285513s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:11.826385247 (5h4m25.413441468s) [upload=5217]: Next segment err: context deadline exceeded Aug 7 21:31:11.826544515 (5h4m25.413600736s) [upload=5217]: Done uploading segments... Aug 7 21:31:11.826568178 (5h4m25.413624399s) [upload=5217]: Upload finished. err: context deadline exceeded Aug 7 21:31:11.924007243 (5h4m25.511063484s) [upload=5227]: Starting upload Aug 7 21:31:11.924116868 (5h4m25.511173108s) [upload=5227]: Done starting upload Aug 7 21:31:11.924293237 (5h4m25.511349458s) [upload=5227]: Uploading segments... Aug 7 21:31:11.924515282 (5h4m25.511571553s) [upload=5227]: Got next segment. Inline: false Aug 7 21:31:11.924580593 (5h4m25.511636844s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.924591073 (5h4m25.511647324s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:41.901407105 (5h4m55.488463316s) [upload=5227]: Next segment err: context deadline exceeded Aug 7 21:31:41.901545804 (5h4m55.488602015s) [upload=5227]: Done uploading segments... Aug 7 21:31:41.901568547 (5h4m55.488624747s) [upload=5227]: Upload finished. err: context deadline exceeded Aug 7 21:32:25.036046257 (5h5m38.623102498s) [upload=5243]: Starting upload Aug 7 21:32:25.036157154 (5h5m38.623213395s) [upload=5243]: Done starting upload Aug 7 21:32:25.036227865 (5h5m38.623284077s) [upload=5243]: Uploading segments... Aug 7 21:32:25.036538426 (5h5m38.623594667s) [upload=5243]: Got next segment. Inline: false Aug 7 21:32:25.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded
Duration: 30.001428861s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000401308s
FAIL (2025-08-07 21:37:22 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g 7 21:30:03.094320953 (5h3m16.681377174s) [upload=5202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:47.146001890 (5h4m0.733058111s) [upload=5219]: Starting upload Aug 7 21:30:47.146103910 (5h4m0.733160131s) [upload=5219]: Done starting upload Aug 7 21:30:47.146153031 (5h4m0.733209242s) [upload=5219]: Uploading segments... Aug 7 21:30:47.146191593 (5h4m0.733247814s) [upload=5219]: Got next segment. Inline: true Aug 7 21:30:47.146196132 (5h4m0.733252343s) [upload=5219]: Waiting for error group managing segments... Aug 7 21:30:47.146229184 (5h4m0.733285395s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:47.146233180 (5h4m0.733289392s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:47.146235815 (5h4m0.733292036s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:17.134124198 (5h4m30.721180449s) [upload=5219]: Done uploading segments... Aug 7 21:31:17.134145378 (5h4m30.721201618s) [upload=5219]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:18.629774505 (5h4m32.216830716s) [upload=5229]: Starting upload Aug 7 21:31:18.629831501 (5h4m32.216887712s) [upload=5229]: Done starting upload Aug 7 21:31:18.629882436 (5h4m32.216938658s) [upload=5229]: Uploading segments... Aug 7 21:31:18.629913235 (5h4m32.216969445s) [upload=5229]: Got next segment. Inline: true Aug 7 21:31:18.629916951 (5h4m32.216973162s) [upload=5229]: Waiting for error group managing segments... Aug 7 21:31:18.629950052 (5h4m32.217006263s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.629953899 (5h4m32.21701012s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:18.629956264 (5h4m32.217012475s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:48.616363445 (5h5m2.203419657s) [upload=5229]: Done uploading segments... Aug 7 21:31:48.616374867 (5h5m2.203431078s) [upload=5229]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:23.393738541 (5h5m36.980794762s) [upload=5242]: Starting upload Aug 7 21:32:23.393794196 (5h5m36.980850406s) [upload=5242]: Done starting upload Aug 7 21:32:23.393846161 (5h5m36.980902382s) [upload=5242]: Uploading segments... Aug 7 21:32:23.393884694 (5h5m36.980940915s) [upload=5242]: Got next segment. Inline: true Aug 7 21:32:23.393888612 (5h5m36.980944822s) [upload=5242]: Waiting for error group managing segments... Aug 7 21:32:23.393921854 (5h5m36.980978075s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:23.393925500 (5h5m36.980981721s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:23.393927785 (5h5m36.980984006s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.381802738 (5h6m6.968858979s) [upload=5242]: Done uploading segments... Aug 7 21:32:53.381824659 (5h6m6.9688809s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000420174s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000902675s
FAIL (2025-08-07 21:37:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000914779s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.00060343s
FAIL (2025-08-07 21:37:15 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.526631147s) [upload=5139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:05.045401846 (5h0m18.632458067s) [upload=5157]: Starting upload Aug 7 21:27:05.045447422 (5h0m18.632503643s) [upload=5157]: Done starting upload Aug 7 21:27:05.045500470 (5h0m18.632556691s) [upload=5157]: Uploading segments... Aug 7 21:27:05.045530256 (5h0m18.632586467s) [upload=5157]: Got next segment. Inline: true Aug 7 21:27:05.045533973 (5h0m18.632590194s) [upload=5157]: Waiting for error group managing segments... Aug 7 21:27:05.045565873 (5h0m18.632622094s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:05.045571213 (5h0m18.632627434s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:05.045573487 (5h0m18.632629708s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:05.013376070 (5h1m18.600432281s) [upload=5157]: Done uploading segments... Aug 7 21:28:05.013391970 (5h1m18.600448181s) [upload=5157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:05.107129855 (5h1m18.694186067s) [upload=5174]: Starting upload Aug 7 21:28:05.107184377 (5h1m18.694240598s) [upload=5174]: Done starting upload Aug 7 21:28:05.107240842 (5h1m18.694297063s) [upload=5174]: Uploading segments... Aug 7 21:28:05.107277301 (5h1m18.694333512s) [upload=5174]: Got next segment. Inline: true Aug 7 21:28:05.107280647 (5h1m18.694336868s) [upload=5174]: Waiting for error group managing segments... Aug 7 21:28:05.107312757 (5h1m18.694368969s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:05.107316373 (5h1m18.694372584s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:05.107318867 (5h1m18.694375088s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:05.085592777 (5h2m18.672648989s) [upload=5174]: Done uploading segments... Aug 7 21:29:05.085611523 (5h2m18.672667733s) [upload=5174]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:05.172798369 (5h2m18.75985459s) [upload=5191]: Starting upload Aug 7 21:29:05.172850326 (5h2m18.759906537s) [upload=5191]: Done starting upload Aug 7 21:29:05.172893617 (5h2m18.759949839s) [upload=5191]: Uploading segments... Aug 7 21:29:05.172921278 (5h2m18.759977499s) [upload=5191]: Got next segment. Inline: true Aug 7 21:29:05.172924825 (5h2m18.759981046s) [upload=5191]: Waiting for error group managing segments... Aug 7 21:29:05.172954671 (5h2m18.760010882s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:05.172958317 (5h2m18.760014538s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:05.172960632 (5h2m18.760016843s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:05.162199693 (5h3m18.749255944s) [upload=5191]: Done uploading segments... Aug 7 21:30:05.162220462 (5h3m18.749276703s) [upload=5191]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:05.259129610 (5h3m18.846185831s) [upload=5205]: Starting upload Aug 7 21:30:05.259178532 (5h3m18.846234742s) [upload=5205]: Done starting upload Aug 7 21:30:05.259225909 (5h3m18.846282131s) [upload=5205]: Uploading segments... Aug 7 21:30:05.259254723 (5h3m18.846310935s) [upload=5205]: Got next segment. Inline: true Aug 7 21:30:05.259258370 (5h3m18.846314591s) [upload=5205]: Waiting for error group managing segments... Aug 7 21:30:05.259289909 (5h3m18.846346131s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:05.259293515 (5h3m18.846349736s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:05.259295659 (5h3m18.84635187s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:05.245621000 (5h4m18.83267725s) [upload=5205]: Done uploading segments... Aug 7 21:31:05.245643692 (5h4m18.832699933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000616194s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.310598038s
FAIL (2025-08-07 21:37:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.31062531s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [......................................X........XX...X.......] duration:
28.905928ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:37:08 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 28.92417ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
33 out of 60 is failed (threshold=3) [.........................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.00110586s
FAIL (2025-08-07 21:37:02 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001131477s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001635151s
FAIL (2025-08-07 21:36:57 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== segment. Inline: false Aug 7 21:28:06.371506767 (5h1m19.958563008s) [upload=5175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:06.371512959 (5h1m19.958569209s) [upload=5175]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:28:36.358597390 (5h1m49.945653632s) [upload=5175]: Next segment err: context deadline exceeded Aug 7 21:28:36.358673913 (5h1m49.945730154s) [upload=5175]: Done uploading segments... Aug 7 21:28:36.358704921 (5h1m49.945761162s) [upload=5175]: Upload finished. err: context deadline exceeded Aug 7 21:29:01.102431480 (5h2m14.689487702s) [upload=5189]: Starting upload Aug 7 21:29:01.102546615 (5h2m14.689602836s) [upload=5189]: Done starting upload Aug 7 21:29:01.102685454 (5h2m14.689741675s) [upload=5189]: Uploading segments... Aug 7 21:29:01.102866763 (5h2m14.689922984s) [upload=5189]: Got next segment. Inline: false Aug 7 21:29:01.102903291 (5h2m14.689959502s) [upload=5189]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:01.102907839 (5h2m14.68996406s) [upload=5189]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:31.087349640 (5h2m44.67440585s) [upload=5189]: Next segment err: context deadline exceeded Aug 7 21:29:31.087429919 (5h2m44.67448613s) [upload=5189]: Done uploading segments... Aug 7 21:29:31.087444957 (5h2m44.674501178s) [upload=5189]: Upload finished. err: context deadline exceeded Aug 7 21:30:07.260405000 (5h3m20.847461241s) [upload=5206]: Starting upload Aug 7 21:30:07.260495098 (5h3m20.84755135s) [upload=5206]: Done starting upload Aug 7 21:30:07.260642223 (5h3m20.847698474s) [upload=5206]: Uploading segments... Aug 7 21:30:07.260927024 (5h3m20.847983276s) [upload=5206]: Got next segment. Inline: false Aug 7 21:30:07.260984843 (5h3m20.848041084s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:07.260994751 (5h3m20.848050993s) [upload=5206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:30:37.232137150 (5h3m50.81919336s) [upload=5206]: Next segment err: context deadline exceeded Aug 7 21:30:37.232307578 (5h3m50.819363789s) [upload=5206]: Done uploading segments... Aug 7 21:30:37.232355838 (5h3m50.819412059s) [upload=5206]: Upload finished. err: context deadline exceeded Aug 7 21:30:41.838696318 (5h3m55.425752559s) [upload=5217]: Starting upload Aug 7 21:30:41.838781527 (5h3m55.425837767s) [upload=5217]: Done starting upload Aug 7 21:30:41.838903785 (5h3m55.425960006s) [upload=5217]: Uploading segments... Aug 7 21:30:41.839179650 (5h3m55.426235901s) [upload=5217]: Got next segment. Inline: false Aug 7 21:30:41.839223772 (5h3m55.426280023s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:41.839229273 (5h3m55.426285513s) [upload=5217]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:11.826385247 (5h4m25.413441468s) [upload=5217]: Next segment err: context deadline exceeded Aug 7 21:31:11.826544515 (5h4m25.413600736s) [upload=5217]: Done uploading segments... Aug 7 21:31:11.826568178 (5h4m25.413624399s) [upload=5217]: Upload finished. err: context deadline exceeded Aug 7 21:31:11.924007243 (5h4m25.511063484s) [upload=5227]: Starting upload Aug 7 21:31:11.924116868 (5h4m25.511173108s) [upload=5227]: Done starting upload Aug 7 21:31:11.924293237 (5h4m25.511349458s) [upload=5227]: Uploading segments... Aug 7 21:31:11.924515282 (5h4m25.511571553s) [upload=5227]: Got next segment. Inline: false Aug 7 21:31:11.924580593 (5h4m25.511636844s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.924591073 (5h4m25.511647324s) [upload=5227]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:41.901407105 (5h4m55.488463316s) [upload=5227]: Next segment err: context deadline exceeded Aug 7 21:31:41.901545804 (5h4m55.488602015s) [upload=5227]: Done uploading segments... Aug 7 21:31:41.901568547 (5h4m55.488624747s) [upload=5227]: Upload finished. err: context deadline exceeded Aug 7 21:32:25.036046257 (5h5m38.623102498s) [upload=5243]: Starting upload Aug 7 21:32:25.036157154 (5h5m38.623213395s) [upload=5243]: Done starting upload Aug 7 21:32:25.036227865 (5h5m38.623284077s) [upload=5243]: Uploading segments... Aug 7 21:32:25.036538426 (5h5m38.623594667s) [upload=5243]: Got next segment. Inline: false Aug 7 21:32:25.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded
Duration: 30.001656351s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001294765s
FAIL (2025-08-07 21:36:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001313199s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000737793s
FAIL (2025-08-07 21:36:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000748304s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
903.598036ms
FAIL (2025-08-07 21:36:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 903.613936ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X........XX...X......] duration:
33.489413ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:36:40 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 33.509991ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.006331548s
FAIL (2025-08-07 21:36:37 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.006352887s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001393216s
FAIL (2025-08-07 21:36:34 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 21:29:33.039722394 (5h2m46.626778635s) [upload=5190]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.107560385 (5h2m46.694616626s) [upload=5202]: Starting upload Aug 7 21:29:33.107638791 (5h2m46.694695032s) [upload=5202]: Done starting upload Aug 7 21:29:33.107697620 (5h2m46.694753871s) [upload=5202]: Uploading segments... Aug 7 21:29:33.107747444 (5h2m46.694803705s) [upload=5202]: Got next segment. Inline: true Aug 7 21:29:33.107753225 (5h2m46.694809465s) [upload=5202]: Waiting for error group managing segments... Aug 7 21:29:33.107796575 (5h2m46.694852816s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.107802446 (5h2m46.694858697s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.107805812 (5h2m46.694862053s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.094304693 (5h3m16.681360904s) [upload=5202]: Done uploading segments... Aug 7 21:30:03.094320953 (5h3m16.681377174s) [upload=5202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:47.146001890 (5h4m0.733058111s) [upload=5219]: Starting upload Aug 7 21:30:47.146103910 (5h4m0.733160131s) [upload=5219]: Done starting upload Aug 7 21:30:47.146153031 (5h4m0.733209242s) [upload=5219]: Uploading segments... Aug 7 21:30:47.146191593 (5h4m0.733247814s) [upload=5219]: Got next segment. Inline: true Aug 7 21:30:47.146196132 (5h4m0.733252343s) [upload=5219]: Waiting for error group managing segments... Aug 7 21:30:47.146229184 (5h4m0.733285395s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:47.146233180 (5h4m0.733289392s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:47.146235815 (5h4m0.733292036s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:17.134124198 (5h4m30.721180449s) [upload=5219]: Done uploading segments... Aug 7 21:31:17.134145378 (5h4m30.721201618s) [upload=5219]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:18.629774505 (5h4m32.216830716s) [upload=5229]: Starting upload Aug 7 21:31:18.629831501 (5h4m32.216887712s) [upload=5229]: Done starting upload Aug 7 21:31:18.629882436 (5h4m32.216938658s) [upload=5229]: Uploading segments... Aug 7 21:31:18.629913235 (5h4m32.216969445s) [upload=5229]: Got next segment. Inline: true Aug 7 21:31:18.629916951 (5h4m32.216973162s) [upload=5229]: Waiting for error group managing segments... Aug 7 21:31:18.629950052 (5h4m32.217006263s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.629953899 (5h4m32.21701012s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:18.629956264 (5h4m32.217012475s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:48.616363445 (5h5m2.203419657s) [upload=5229]: Done uploading segments... Aug 7 21:31:48.616374867 (5h5m2.203431078s) [upload=5229]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:23.393738541 (5h5m36.980794762s) [upload=5242]: Starting upload Aug 7 21:32:23.393794196 (5h5m36.980850406s) [upload=5242]: Done starting upload Aug 7 21:32:23.393846161 (5h5m36.980902382s) [upload=5242]: Uploading segments... Aug 7 21:32:23.393884694 (5h5m36.980940915s) [upload=5242]: Got next segment. Inline: true Aug 7 21:32:23.393888612 (5h5m36.980944822s) [upload=5242]: Waiting for error group managing segments... Aug 7 21:32:23.393921854 (5h5m36.980978075s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:23.393925500 (5h5m36.980981721s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:23.393927785 (5h5m36.980984006s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.381802738 (5h6m6.968858979s) [upload=5242]: Done uploading segments... Aug 7 21:32:53.381824659 (5h6m6.9688809s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001404116s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
23 out of 60 is failed (threshold=3) [....................................XX.XXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001016291s
FAIL (2025-08-07 21:36:33 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001031209s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X...X...] duration:
112.710373ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:36:29 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 112.736992ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
32 out of 60 is failed (threshold=3) [..........................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001047672s
FAIL (2025-08-07 21:36:25 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001059823s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration:
1m0.000806012s
FAIL (2025-08-07 21:36:17 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
) [noise=true , address=104.36.82.8:4987 , piece_id=UT4GNLD5ASIMDPTXPRDMY2PLQMTISUWWKKKD4Y5ZXZ4BQ44KHNIQ, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.466247081 (4h55m45.053303293s) [noise=true , address=104.36.82.35:4589 , piece_id=2TNQCRWILNH3LK5DL6KHOUWVRDD3VFQKTTJI43QZTEZ2H2QU6SUA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.467832441 (4h55m45.054888662s) [noise=true , address=45.91.116.8:4065 , piece_id=RUAZW4OEX5ICVVC25Y2SXGXUMSYAYGGBASTKT35ISYCVCOP7BCZQ, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.474570919 (4h55m45.06162715s) [noise=true , address=104.36.82.17:4353 , piece_id=N5HNMPS45SEQRALZ7XT6FMHZFTS7FHFH24FXIOMVFC6VTSSRE2ZA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.476328280 (4h55m45.063384491s) [noise=true , address=104.36.82.15:4286 , piece_id=GAUDF7WSKVIEPTQAI6EGGIU336B2JRRJPNL6OY7KX3AFFDNKBBYA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.476661793 (4h55m45.063718024s) [noise=true , address=104.36.82.60:4367 , piece_id=MJA4TVZARNCFUTOSLH6KWSIP7ND4RKYD6OH7KYT3OFT37K2MANGA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.486513301 (4h55m45.073569513s) [noise=true , address=104.36.82.56:4238 , piece_id=J5FYG5FAUX6WDIZDE6YQR6HHVTNREUCY3CCRQVG4AJIJO2ZY33BQ, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.487495355 (4h55m45.074551587s) [noise=true , address=104.36.82.55:4217 , piece_id=NVC2PGB6PAUW3FVTDK6QEGO5B5VF53HKFROP2XDOBFTZOYZJGOZQ, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.488952606 (4h55m45.076008827s) [noise=true , address=104.36.82.16:4310 , piece_id=SBJICEMYSYAH53KD6MEYM7OCWSSWZYX46BSFADPNZZ4MPM6Z7HUQ, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.492385343 (4h55m45.079441574s) [noise=true , address=104.36.82.35:4592 , piece_id=YIERBYJGLQFGTRSOT4ZSJKO62MNDH3DUOAS65434RB2MSBX3EGOA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.504789768 (4h55m45.091845999s) [noise=true , address=104.36.82.5:4864 , piece_id=57RIVHQZ3KKNP5SWWVO4J6L5QGTU24P664SYEYF5BBYCA62Z2GPA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.534017416 (4h55m45.121073647s) [noise=true , address=104.36.82.44:4901 , piece_id=4GQO4GJXY7V3XUHMH7HIG72NO43OEIVO5WZWJRDKOOHCOVU2UF2Q, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.594014387 (4h55m45.181070598s) [noise=true , address=79.127.201.215:4344 , piece_id=S7JKJHCXKZHEHF5YQA7F47A73ZLUXQRMJ3WGVXTQDSBFNO4OPMOA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: <nil>
Aug 7 21:22:31.594041378 (4h55m45.181097599s) [seg_pos={0 0}, upload=5078]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:22:31.594296384 (4h55m45.181352596s) [noise=true , address=104.36.82.37:4673 , piece_id=BSVONXWWBJHRDK7PZV7RYZMF4VCWG2LKKUFRBKZ6WYB4QLDN3ANA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NkBjmh5iL2BtckR19qgXQ4wnfvnbpA9erSuMCTEijwX7ZdHGL): protocol: write tcp 5.161.89.252:24122->104.36.82.37:4673: use of closed network connection; write tcp 5.161.89.252:24122->104.36.82.37:4673: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:24122->104.36.82.37:4673: use of closed network connection
Aug 7 21:22:31.594425967 (4h55m45.181482207s) [noise=true , address=45.91.116.7:4031 , piece_id=RQO5CO4GIRLZH7ZV3PNBNA5CITIR4WPDBX2ANKC3QZIO4277NFAA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW): protocol: write tcp 5.161.89.252:42394->45.91.116.7:4031: use of closed network connection; write tcp 5.161.89.252:42394->45.91.116.7:4031: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:42394->45.91.116.7:4031: use of closed network connection
Aug 7 21:22:31.594492841 (4h55m45.181549062s) [noise=true , address=64.49.13.27:4961 , piece_id=YFFLF6O7U4AGW3TAIQXLDGO4GITRL2PH5LK3WPQRJ5FIOK2UZTHA, seg_pos={0 0}, upload=5078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwCHMdLXowSBhg3UtVahaWWAQ5TJYZ7yUSdAQ8Q8Cv3qWwMnt): protocol: read tcp 5.161.89.252:50748->64.49.13.27:4961: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:22:31.594530211 (4h55m45.181586432s) [noise=true , address=104.36.82.12:4163 , piece_id=G3HPBKE766GXCXFCDJS36BCPGCHA4OVFK32VKVOV4NQGJKGIK67Q, seg_pos={0 0}, upload=5078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC): protocol: read tcp 5.161.89.252:58420->104.36.82.12:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:22:31.594553194 (4h55m45.181609405s) [seg_pos={0 0}, upload=5078]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:22:31.594607895 (4h55m45.181664116s) [upload=5078]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:22:31.594611643 (4h55m45.181667864s) [upload=5078]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:31:08.619701228 (5h4m22.20675747s) [upload=5078]: Done uploading segments...
Aug 7 21:31:08.619736956 (5h4m22.206793196s) [upload=5078]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:31:08.702890161 (5h4m22.289946372s) [upload=5225]: Starting upload
Aug 7 21:31:08.702939665 (5h4m22.289995875s) [upload=5225]: Done starting upload
Aug 7 21:31:08.702997221 (5h4m22.290053433s) [upload=5225]: Uploading segments...
Aug 7 21:31:08.703166718 (5h4m22.290222939s) [upload=5225]: Got next segment. Inline: false
Aug 7 21:31:08.703192467 (5h4m22.290248678s) [upload=5225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:31:08.703196053 (5h4m22.290252264s) [upload=5225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:32:08.682639677 (5h5m22.269695928s) [upload=5225]: Next segment err: context deadline exceeded
Aug 7 21:32:08.682751946 (5h5m22.269808187s) [upload=5225]: Done uploading segments...
Aug 7 21:32:08.682783364 (5h5m22.269839605s) [upload=5225]: Upload finished. err: context deadline exceeded
Aug 7 21:32:08.787326339 (5h5m22.374382581s) [upload=5238]: Starting upload
Aug 7 21:32:08.787416938 (5h5m22.37447319s) [upload=5238]: Done starting upload
Aug 7 21:32:08.787498991 (5h5m22.374555192s) [upload=5238]: Uploading segments...
Aug 7 21:32:08.787790025 (5h5m22.374846276s) [upload=5238]: Got next segment. Inline: false
Aug 7 21:32:08.787837994 (5h5m22.374894235s) [upload=5238]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:32:08.787843474 (5h5m22.374899715s) [upload=5238]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:33:08.772132372 (5h6m22.359188613s) [upload=5238]: Next segment err: context deadline exceeded
Aug 7 21:33:08.772297010 (5h6m22.359353261s) [upload=5238]: Done uploading segments...
Aug 7 21:33:08.772322768 (5h6m22.359379019s) [upload=5238]: Upload finished. err: context deadline exceeded
Aug 7 21:33:16.547013519 (5h6m30.13406973s) [upload=5258]: Starting upload
Aug 7 21:33:16.547055758 (5h6m30.13411198s) [upload=5258]: Done starting upload
Aug 7 21:33:16.547276520 (5h6m30.134332731s) [upload=5258]: Uploading segments...
Aug 7 21:33:16.547303270 (5h6m30.134359492s) [upload=5258]: Got next segment. Inline: false
Aug 7 21:33:16.547325551 (5h6m30.134381772s) [upload=5258]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:33:16.547329078 (5h6m30.134385289s) [upload=5258]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:16.531560242 (5h7m30.118616463s) [upload=5258]: Next segment err: context deadline exceeded
Aug 7 21:34:16.531648257 (5h7m30.118704478s) [upload=5258]: Done uploading segments...
Aug 7 21:34:16.531667602 (5h7m30.118723823s) [upload=5258]: Upload finished. err: context deadline exceeded
Aug 7 21:34:17.876384791 (5h7m31.463441042s) [upload=5276]: Starting upload
Aug 7 21:34:17.876470201 (5h7m31.463526442s) [upload=5276]: Done starting upload
Aug 7 21:34:17.876556292 (5h7m31.463612532s) [upload=5276]: Uploading segments...
Aug 7 21:34:17.876831023 (5h7m31.463887265s) [upload=5276]: Got next segment. Inline: false
Aug 7 21:34:17.876868654 (5h7m31.463924896s) [upload=5276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:17.876873824 (5h7m31.463930065s) [upload=5276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:35:17.864456308 (5h8m31.451512549s) [upload=5276]: Next segment err: context deadline exceeded
Aug 7 21:35:17.864571123 (5h8m31.451627363s) [upload=5276]: Done uploading segments...
Aug 7 21:35:17.864593234 (5h8m31.451649475s) [upload=5276]: Upload finished. err: context deadline exceeded
Aug 7 21:35:17.979672025 (5h8m31.566728237s) [upload=5295]: Starting upload
Aug 7 21:35:17.979730714 (5h8m31.566786916s) [upload=5295]: Done starting upload
Aug 7 21:35:17.979835169 (5h8m31.566891421s) [upload=5295]: Uploading segments...
Aug 7 21:35:17.979956065 (5h8m31.567012306s) [upload=5295]: Got next segment. Inline: false
Aug 7 21:35:17.980001841 (5h8m31.567058082s) [upload=5295]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:35:17.980008273 (5h8m31.567064514s) [upload=5295]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:36:17.942910542 (5h9m31.529966792s) [upload=5295]: Next segment err: context deadline exceeded
Aug 7 21:36:17.943028853 (5h9m31.530085093s) [upload=5295]: Done uploading segments...
Aug 7 21:36:17.943094875 (5h9m31.530151117s) [upload=5295]: Upload finished. err: context deadline exceeded
Duration: 1m0.000822473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.166422514s
FAIL (2025-08-07 21:36:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.16644711s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001339309s
FAIL (2025-08-07 21:36:15 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 14037s) [upload=5127]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:04.958090745 (4h59m18.545146957s) [upload=5139]: Starting upload Aug 7 21:26:04.958138415 (4h59m18.545194626s) [upload=5139]: Done starting upload Aug 7 21:26:04.958184982 (4h59m18.545241203s) [upload=5139]: Uploading segments... Aug 7 21:26:04.958220368 (4h59m18.545276589s) [upload=5139]: Got next segment. Inline: true Aug 7 21:26:04.958224295 (4h59m18.545280516s) [upload=5139]: Waiting for error group managing segments... Aug 7 21:26:04.958253820 (4h59m18.545310031s) [upload=5139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:04.958257427 (4h59m18.545313638s) [upload=5139]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:04.958259772 (4h59m18.545315993s) [upload=5139]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:04.939553376 (5h0m18.526609617s) [upload=5139]: Done uploading segments... Aug 7 21:27:04.939574906 (5h0m18.526631147s) [upload=5139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:05.045401846 (5h0m18.632458067s) [upload=5157]: Starting upload Aug 7 21:27:05.045447422 (5h0m18.632503643s) [upload=5157]: Done starting upload Aug 7 21:27:05.045500470 (5h0m18.632556691s) [upload=5157]: Uploading segments... Aug 7 21:27:05.045530256 (5h0m18.632586467s) [upload=5157]: Got next segment. Inline: true Aug 7 21:27:05.045533973 (5h0m18.632590194s) [upload=5157]: Waiting for error group managing segments... Aug 7 21:27:05.045565873 (5h0m18.632622094s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:05.045571213 (5h0m18.632627434s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:05.045573487 (5h0m18.632629708s) [upload=5157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:05.013376070 (5h1m18.600432281s) [upload=5157]: Done uploading segments... Aug 7 21:28:05.013391970 (5h1m18.600448181s) [upload=5157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:05.107129855 (5h1m18.694186067s) [upload=5174]: Starting upload Aug 7 21:28:05.107184377 (5h1m18.694240598s) [upload=5174]: Done starting upload Aug 7 21:28:05.107240842 (5h1m18.694297063s) [upload=5174]: Uploading segments... Aug 7 21:28:05.107277301 (5h1m18.694333512s) [upload=5174]: Got next segment. Inline: true Aug 7 21:28:05.107280647 (5h1m18.694336868s) [upload=5174]: Waiting for error group managing segments... Aug 7 21:28:05.107312757 (5h1m18.694368969s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:05.107316373 (5h1m18.694372584s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:05.107318867 (5h1m18.694375088s) [upload=5174]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:05.085592777 (5h2m18.672648989s) [upload=5174]: Done uploading segments... Aug 7 21:29:05.085611523 (5h2m18.672667733s) [upload=5174]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:05.172798369 (5h2m18.75985459s) [upload=5191]: Starting upload Aug 7 21:29:05.172850326 (5h2m18.759906537s) [upload=5191]: Done starting upload Aug 7 21:29:05.172893617 (5h2m18.759949839s) [upload=5191]: Uploading segments... Aug 7 21:29:05.172921278 (5h2m18.759977499s) [upload=5191]: Got next segment. Inline: true Aug 7 21:29:05.172924825 (5h2m18.759981046s) [upload=5191]: Waiting for error group managing segments... Aug 7 21:29:05.172954671 (5h2m18.760010882s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:05.172958317 (5h2m18.760014538s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:05.172960632 (5h2m18.760016843s) [upload=5191]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:05.162199693 (5h3m18.749255944s) [upload=5191]: Done uploading segments... Aug 7 21:30:05.162220462 (5h3m18.749276703s) [upload=5191]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:05.259129610 (5h3m18.846185831s) [upload=5205]: Starting upload Aug 7 21:30:05.259178532 (5h3m18.846234742s) [upload=5205]: Done starting upload Aug 7 21:30:05.259225909 (5h3m18.846282131s) [upload=5205]: Uploading segments... Aug 7 21:30:05.259254723 (5h3m18.846310935s) [upload=5205]: Got next segment. Inline: true Aug 7 21:30:05.259258370 (5h3m18.846314591s) [upload=5205]: Waiting for error group managing segments... Aug 7 21:30:05.259289909 (5h3m18.846346131s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:05.259293515 (5h3m18.846349736s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:05.259295659 (5h3m18.84635187s) [upload=5205]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:05.245621000 (5h4m18.83267725s) [upload=5205]: Done uploading segments... Aug 7 21:31:05.245643692 (5h4m18.832699933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001364797s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000707407s
FAIL (2025-08-07 21:36:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.0007246s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration:
120.338308ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:36:07 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 120.371581ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000435418s
FAIL (2025-08-07 21:36:04 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :28:53.132205376 (5h2m6.719261617s) [upload=5182]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:03.059565675 (5h2m16.646621886s) [upload=5190]: Starting upload Aug 7 21:29:03.059620448 (5h2m16.646676659s) [upload=5190]: Done starting upload Aug 7 21:29:03.059662505 (5h2m16.646718727s) [upload=5190]: Uploading segments... Aug 7 21:29:03.059702511 (5h2m16.646758732s) [upload=5190]: Got next segment. Inline: true Aug 7 21:29:03.059707350 (5h2m16.646763561s) [upload=5190]: Waiting for error group managing segments... Aug 7 21:29:03.059743357 (5h2m16.646799578s) [upload=5190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:03.059748677 (5h2m16.646804898s) [upload=5190]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:03.059752263 (5h2m16.646808474s) [upload=5190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:33.039700593 (5h2m46.626756834s) [upload=5190]: Done uploading segments... Aug 7 21:29:33.039722394 (5h2m46.626778635s) [upload=5190]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.107560385 (5h2m46.694616626s) [upload=5202]: Starting upload Aug 7 21:29:33.107638791 (5h2m46.694695032s) [upload=5202]: Done starting upload Aug 7 21:29:33.107697620 (5h2m46.694753871s) [upload=5202]: Uploading segments... Aug 7 21:29:33.107747444 (5h2m46.694803705s) [upload=5202]: Got next segment. Inline: true Aug 7 21:29:33.107753225 (5h2m46.694809465s) [upload=5202]: Waiting for error group managing segments... Aug 7 21:29:33.107796575 (5h2m46.694852816s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.107802446 (5h2m46.694858697s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.107805812 (5h2m46.694862053s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.094304693 (5h3m16.681360904s) [upload=5202]: Done uploading segments... Aug 7 21:30:03.094320953 (5h3m16.681377174s) [upload=5202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:47.146001890 (5h4m0.733058111s) [upload=5219]: Starting upload Aug 7 21:30:47.146103910 (5h4m0.733160131s) [upload=5219]: Done starting upload Aug 7 21:30:47.146153031 (5h4m0.733209242s) [upload=5219]: Uploading segments... Aug 7 21:30:47.146191593 (5h4m0.733247814s) [upload=5219]: Got next segment. Inline: true Aug 7 21:30:47.146196132 (5h4m0.733252343s) [upload=5219]: Waiting for error group managing segments... Aug 7 21:30:47.146229184 (5h4m0.733285395s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:47.146233180 (5h4m0.733289392s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:47.146235815 (5h4m0.733292036s) [upload=5219]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:17.134124198 (5h4m30.721180449s) [upload=5219]: Done uploading segments... Aug 7 21:31:17.134145378 (5h4m30.721201618s) [upload=5219]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:18.629774505 (5h4m32.216830716s) [upload=5229]: Starting upload Aug 7 21:31:18.629831501 (5h4m32.216887712s) [upload=5229]: Done starting upload Aug 7 21:31:18.629882436 (5h4m32.216938658s) [upload=5229]: Uploading segments... Aug 7 21:31:18.629913235 (5h4m32.216969445s) [upload=5229]: Got next segment. Inline: true Aug 7 21:31:18.629916951 (5h4m32.216973162s) [upload=5229]: Waiting for error group managing segments... Aug 7 21:31:18.629950052 (5h4m32.217006263s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.629953899 (5h4m32.21701012s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:18.629956264 (5h4m32.217012475s) [upload=5229]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:48.616363445 (5h5m2.203419657s) [upload=5229]: Done uploading segments... Aug 7 21:31:48.616374867 (5h5m2.203431078s) [upload=5229]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:23.393738541 (5h5m36.980794762s) [upload=5242]: Starting upload Aug 7 21:32:23.393794196 (5h5m36.980850406s) [upload=5242]: Done starting upload Aug 7 21:32:23.393846161 (5h5m36.980902382s) [upload=5242]: Uploading segments... Aug 7 21:32:23.393884694 (5h5m36.980940915s) [upload=5242]: Got next segment. Inline: true Aug 7 21:32:23.393888612 (5h5m36.980944822s) [upload=5242]: Waiting for error group managing segments... Aug 7 21:32:23.393921854 (5h5m36.980978075s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:23.393925500 (5h5m36.980981721s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:23.393927785 (5h5m36.980984006s) [upload=5242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.381802738 (5h6m6.968858979s) [upload=5242]: Done uploading segments... Aug 7 21:32:53.381824659 (5h6m6.9688809s) [upload=5242]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:22.334593983 (5h6m35.921650204s) [upload=5260]: Starting upload Aug 7 21:33:22.334631033 (5h6m35.921687254s) [upload=5260]: Done starting upload Aug 7 21:33:22.334676617 (5h6m35.921732838s) [upload=5260]: Uploading segments... Aug 7 21:33:22.334702015 (5h6m35.921758236s) [upload=5260]: Got next segment. Inline: true Aug 7 21:33:22.334705391 (5h6m35.921761602s) [upload=5260]: Waiting for error group managing segments... Aug 7 21:33:22.334736159 (5h6m35.92179237s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:22.334739706 (5h6m35.921795927s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:22.334742220 (5h6m35.921798431s) [upload=5260]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:52.320399577 (5h7m5.907455798s) [upload=5260]: Done uploading segments... Aug 7 21:33:52.320416559 (5h7m5.90747278s) [upload=5260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000446729s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
31 out of 60 is failed (threshold=3) [...........................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001974493s
FAIL (2025-08-07 21:35:55 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001995121s
core
uplink
us1
|