Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.000690943s FAIL (2025-08-07 21:29:05 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.189934289 (4h54m43.776990511s) [upload=5071]: Starting upload Aug 7 21:21:30.189980476 (4h54m43.777036687s) [upload=5071]: Done starting upload Aug 7 21:21:30.190017424 (4h54m43.777073645s) [upload=5071]: Uploading segments... Aug 7 21:21:30.190043793 (4h54m43.777100005s) [upload=5071]: Got next segment. Inline: true Aug 7 21:21:30.190047100 (4h54m43.777103321s) [upload=5071]: Waiting for error group managing segments... Aug 7 21:21:30.190088458 (4h54m43.777144669s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.190092425 (4h54m43.777148646s) [upload=5071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.190094709 (4h54m43.77715093s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.179283539 (4h55m43.76633979s) [upload=5071]: Done uploading segments... Aug 7 21:22:30.179305229 (4h55m43.766361471s) [upload=5071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.041348848 (4h55m57.628405089s) [upload=5090]: Starting upload Aug 7 21:22:44.041428967 (4h55m57.628485208s) [upload=5090]: Done starting upload Aug 7 21:22:44.041486454 (4h55m57.628542695s) [upload=5090]: Uploading segments... Aug 7 21:22:44.041542930 (4h55m57.628599171s) [upload=5090]: Got next segment. Inline: true Aug 7 21:22:44.041549963 (4h55m57.628606204s) [upload=5090]: Waiting for error group managing segments... Aug 7 21:22:44.041596389 (4h55m57.628652631s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.041602170 (4h55m57.628658411s) [upload=5090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.041605667 (4h55m57.628661917s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.025768658 (4h56m57.612824879s) [upload=5090]: Done uploading segments... Aug 7 21:23:44.025784877 (4h56m57.612841089s) [upload=5090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:58.208173683 (4h57m11.795229924s) [upload=5112]: Starting upload Aug 7 21:23:58.208257669 (4h57m11.795313911s) [upload=5112]: Done starting upload Aug 7 21:23:58.208337199 (4h57m11.79539344s) [upload=5112]: Uploading segments... Aug 7 21:23:58.208378025 (4h57m11.795434265s) [upload=5112]: Got next segment. Inline: true Aug 7 21:23:58.208383675 (4h57m11.795439916s) [upload=5112]: Waiting for error group managing segments... Aug 7 21:23:58.208427437 (4h57m11.795483678s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:58.208433278 (4h57m11.795489519s) [upload=5112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:58.208438388 (4h57m11.795494628s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:58.195766004 (4h58m11.782822225s) [upload=5112]: Done uploading segments... Aug 7 21:24:58.195780731 (4h58m11.782836943s) [upload=5112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:04.854742735 (4h58m18.441798946s) [upload=5127]: Starting upload Aug 7 21:25:04.854807696 (4h58m18.441863907s) [upload=5127]: Done starting upload Aug 7 21:25:04.854870773 (4h58m18.441926984s) [upload=5127]: Uploading segments... Aug 7 21:25:04.854916199 (4h58m18.44197242s) [upload=5127]: Got next segment. Inline: true Aug 7 21:25:04.854936166 (4h58m18.441992387s) [upload=5127]: Waiting for error group managing segments... Aug 7 21:25:04.854972444 (4h58m18.442028655s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:04.854976711 (4h58m18.442032922s) [upload=5127]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:04.854979997 (4h58m18.442036218s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:04.841433540 (4h59m18.428489761s) [upload=5127]: Done uploading segments... Aug 7 21:26:04.841457825 (4h59m18.428514037s) [upload=5127]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:04.958090745 (4h59m18.545146957s) [upload=5139]: Starting upload Aug 7 21:26:04.958138415 (4h59m18.545194626s) [upload=5139]: Done starting upload Aug 7 21:26:04.958184982 (4h59m18.545241203s) [upload=5139]: Uploading segments... Aug 7 21:26:04.958220368 (4h59m18.545276589s) [upload=5139]: Got next segment. Inline: true Aug 7 21:26:04.958224295 (4h59m18.545280516s) [upload=5139]: Waiting for error group managing segments... Aug 7 21:26:04.958253820 (4h59m18.545310031s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:04.958257427 (4h59m18.545313638s) [upload=5139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:04.958259772 (4h59m18.545315993s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:04.939553376 (5h0m18.526609617s) [upload=5139]: Done uploading segments... Aug 7 21:27:04.939574906 (5h0m18.526631147s) [upload=5139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:05.045401846 (5h0m18.632458067s) [upload=5157]: Starting upload Aug 7 21:27:05.045447422 (5h0m18.632503643s) [upload=5157]: Done starting upload Aug 7 21:27:05.045500470 (5h0m18.632556691s) [upload=5157]: Uploading segments... Aug 7 21:27:05.045530256 (5h0m18.632586467s) [upload=5157]: Got next segment. Inline: true Aug 7 21:27:05.045533973 (5h0m18.632590194s) [upload=5157]: Waiting for error group managing segments... Aug 7 21:27:05.045565873 (5h0m18.632622094s) [upload=5157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:05.045571213 (5h0m18.632627434s) [upload=5157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:05.045573487 (5h0m18.632629708s) [upload=5157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:05.013376070 (5h1m18.600432281s) [upload=5157]: Done uploading segments... Aug 7 21:28:05.013391970 (5h1m18.600448181s) [upload=5157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:05.107129855 (5h1m18.694186067s) [upload=5174]: Starting upload Aug 7 21:28:05.107184377 (5h1m18.694240598s) [upload=5174]: Done starting upload Aug 7 21:28:05.107240842 (5h1m18.694297063s) [upload=5174]: Uploading segments... Aug 7 21:28:05.107277301 (5h1m18.694333512s) [upload=5174]: Got next segment. Inline: true Aug 7 21:28:05.107280647 (5h1m18.694336868s) [upload=5174]: Waiting for error group managing segments... Aug 7 21:28:05.107312757 (5h1m18.694368969s) [upload=5174]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:05.107316373 (5h1m18.694372584s) [upload=5174]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:05.107318867 (5h1m18.694375088s) [upload=5174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:05.085592777 (5h2m18.672648989s) [upload=5174]: Done uploading segments... Aug 7 21:29:05.085611523 (5h2m18.672667733s) [upload=5174]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000702987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.000579235s FAIL (2025-08-07 21:28:53 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== text deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:27.243732093 (4h58m40.830788304s) [upload=5132]: Starting upload Aug 7 21:25:27.243787377 (4h58m40.830843598s) [upload=5132]: Done starting upload Aug 7 21:25:27.243846396 (4h58m40.830902607s) [upload=5132]: Uploading segments... Aug 7 21:25:27.243883505 (4h58m40.830939727s) [upload=5132]: Got next segment. Inline: true Aug 7 21:25:27.243887303 (4h58m40.830943524s) [upload=5132]: Waiting for error group managing segments... Aug 7 21:25:27.243942507 (4h58m40.830998728s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:27.243946624 (4h58m40.831002845s) [upload=5132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:27.243948898 (4h58m40.831005119s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.223887428 (4h59m10.810943669s) [upload=5132]: Done uploading segments... Aug 7 21:25:57.223909498 (4h59m10.81096574s) [upload=5132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:29.142150756 (4h59m42.729206977s) [upload=5147]: Starting upload Aug 7 21:26:29.142194398 (4h59m42.729250619s) [upload=5147]: Done starting upload Aug 7 21:26:29.142233742 (4h59m42.729289963s) [upload=5147]: Uploading segments... Aug 7 21:26:29.142260401 (4h59m42.729316612s) [upload=5147]: Got next segment. Inline: true Aug 7 21:26:29.142263868 (4h59m42.729320089s) [upload=5147]: Waiting for error group managing segments... Aug 7 21:26:29.142293002 (4h59m42.729349213s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.142297080 (4h59m42.729353301s) [upload=5147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:29.142299354 (4h59m42.729355575s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:59.127042625 (5h0m12.714098866s) [upload=5147]: Done uploading segments... Aug 7 21:26:59.127117424 (5h0m12.714173665s) [upload=5147]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:10.814290180 (5h0m24.401346421s) [upload=5160]: Starting upload Aug 7 21:27:10.814371522 (5h0m24.401427764s) [upload=5160]: Done starting upload Aug 7 21:27:10.814438597 (5h0m24.401494839s) [upload=5160]: Uploading segments... Aug 7 21:27:10.814480476 (5h0m24.401536717s) [upload=5160]: Got next segment. Inline: true Aug 7 21:27:10.814485826 (5h0m24.401542067s) [upload=5160]: Waiting for error group managing segments... Aug 7 21:27:10.814529677 (5h0m24.401585918s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:10.814535198 (5h0m24.401591438s) [upload=5160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:10.814538994 (5h0m24.401595236s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:40.714926118 (5h0m54.301982339s) [upload=5160]: Done uploading segments... Aug 7 21:27:40.714942067 (5h0m54.301998278s) [upload=5160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:40.796846333 (5h0m54.383902554s) [upload=5169]: Starting upload Aug 7 21:27:40.796900224 (5h0m54.383956435s) [upload=5169]: Done starting upload Aug 7 21:27:40.796945107 (5h0m54.384001318s) [upload=5169]: Uploading segments... Aug 7 21:27:40.796979782 (5h0m54.384036003s) [upload=5169]: Got next segment. Inline: true Aug 7 21:27:40.796983369 (5h0m54.38403959s) [upload=5169]: Waiting for error group managing segments... Aug 7 21:27:40.797013695 (5h0m54.384069907s) [upload=5169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:40.797017642 (5h0m54.384073863s) [upload=5169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:40.797020007 (5h0m54.384076218s) [upload=5169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:10.771421400 (5h1m24.358477651s) [upload=5169]: Done uploading segments... Aug 7 21:28:10.771479779 (5h1m24.35853602s) [upload=5169]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:23.151092147 (5h1m36.738148388s) [upload=5182]: Starting upload Aug 7 21:28:23.151190951 (5h1m36.738247193s) [upload=5182]: Done starting upload Aug 7 21:28:23.151342555 (5h1m36.738398796s) [upload=5182]: Uploading segments... Aug 7 21:28:23.151386697 (5h1m36.738442937s) [upload=5182]: Got next segment. Inline: true Aug 7 21:28:23.151392588 (5h1m36.738448829s) [upload=5182]: Waiting for error group managing segments... Aug 7 21:28:23.151436370 (5h1m36.738492611s) [upload=5182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:23.151441740 (5h1m36.738497981s) [upload=5182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:23.151445377 (5h1m36.738501618s) [upload=5182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:53.132183214 (5h2m6.719239455s) [upload=5182]: Done uploading segments... Aug 7 21:28:53.132205376 (5h2m6.719261617s) [upload=5182]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000602428s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.000459264s FAIL (2025-08-07 21:28:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000484419s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.001219262s FAIL (2025-08-07 21:28:40 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001234851s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.00137679s FAIL (2025-08-07 21:28:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Aug 7 21:25:15.571225478 (4h58m29.158281699s) [upload=5129]: Starting upload Aug 7 21:25:15.571281894 (4h58m29.158338105s) [upload=5129]: Done starting upload Aug 7 21:25:15.571359468 (4h58m29.158415689s) [upload=5129]: Uploading segments... Aug 7 21:25:15.571564180 (4h58m29.158620401s) [upload=5129]: Got next segment. Inline: false Aug 7 21:25:15.571622810 (4h58m29.158679021s) [upload=5129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.571628479 (4h58m29.1586847s) [upload=5129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:45.547010083 (4h58m59.134066294s) [upload=5129]: Next segment err: context deadline exceeded Aug 7 21:25:45.547084462 (4h58m59.134140683s) [upload=5129]: Done uploading segments... Aug 7 21:25:45.547107044 (4h58m59.134163266s) [upload=5129]: Upload finished. err: context deadline exceeded Aug 7 21:25:45.641521062 (4h58m59.228577283s) [upload=5137]: Starting upload Aug 7 21:25:45.641578831 (4h58m59.228635041s) [upload=5137]: Done starting upload Aug 7 21:25:45.641631238 (4h58m59.228687459s) [upload=5137]: Uploading segments... Aug 7 21:25:45.641794333 (4h58m59.228850554s) [upload=5137]: Got next segment. Inline: false Aug 7 21:25:45.641818608 (4h58m59.228874829s) [upload=5137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.641822736 (4h58m59.228878947s) [upload=5137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:15.630688562 (4h59m29.217744793s) [upload=5137]: Next segment err: context deadline exceeded Aug 7 21:26:15.630873647 (4h59m29.217929858s) [upload=5137]: Done uploading segments... Aug 7 21:26:15.630892703 (4h59m29.217948914s) [upload=5137]: Upload finished. err: context deadline exceeded Aug 7 21:26:29.805240268 (4h59m43.392296479s) [upload=5148]: Starting upload Aug 7 21:26:29.805289190 (4h59m43.392345411s) [upload=5148]: Done starting upload Aug 7 21:26:29.805393004 (4h59m43.392449225s) [upload=5148]: Uploading segments... Aug 7 21:26:29.805516605 (4h59m43.392572816s) [upload=5148]: Got next segment. Inline: false Aug 7 21:26:29.805549756 (4h59m43.392605967s) [upload=5148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.805553633 (4h59m43.392609854s) [upload=5148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:59.790298417 (5h0m13.377354628s) [upload=5148]: Next segment err: context deadline exceeded Aug 7 21:26:59.790450801 (5h0m13.377507012s) [upload=5148]: Done uploading segments... Aug 7 21:26:59.790479374 (5h0m13.377535595s) [upload=5148]: Upload finished. err: context deadline exceeded Aug 7 21:26:59.854535904 (5h0m13.441592145s) [upload=5156]: Starting upload Aug 7 21:26:59.854627225 (5h0m13.441683465s) [upload=5156]: Done starting upload Aug 7 21:26:59.854723153 (5h0m13.441779394s) [upload=5156]: Uploading segments... Aug 7 21:26:59.855007455 (5h0m13.442063705s) [upload=5156]: Got next segment. Inline: false Aug 7 21:26:59.855059541 (5h0m13.442115782s) [upload=5156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:59.855096871 (5h0m13.442153123s) [upload=5156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:29.841757946 (5h0m43.428814187s) [upload=5156]: Next segment err: context deadline exceeded Aug 7 21:27:29.841960274 (5h0m43.429016514s) [upload=5156]: Done uploading segments... Aug 7 21:27:29.842007351 (5h0m43.429063593s) [upload=5156]: Upload finished. err: context deadline exceeded Aug 7 21:28:06.370955849 (5h1m19.95801209s) [upload=5175]: Starting upload Aug 7 21:28:06.371030067 (5h1m19.958086308s) [upload=5175]: Done starting upload Aug 7 21:28:06.371151734 (5h1m19.958207975s) [upload=5175]: Uploading segments... Aug 7 21:28:06.371451254 (5h1m19.958507495s) [upload=5175]: Got next segment. Inline: false Aug 7 21:28:06.371506767 (5h1m19.958563008s) [upload=5175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:06.371512959 (5h1m19.958569209s) [upload=5175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:36.358597390 (5h1m49.945653632s) [upload=5175]: Next segment err: context deadline exceeded Aug 7 21:28:36.358673913 (5h1m49.945730154s) [upload=5175]: Done uploading segments... Aug 7 21:28:36.358704921 (5h1m49.945761162s) [upload=5175]: Upload finished. err: context deadline exceeded Duration: 30.001399702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.....................................XX..XXXXXXXXXXXXXXXXXXX] duration: 30.000815143s FAIL (2025-08-07 21:28:32 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000833457s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.000795776s FAIL (2025-08-07 21:28:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00083514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [............................................XX.XXXXXXXXXXXXX] duration: 1m0.000771649s FAIL (2025-08-07 21:28:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000794061s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.001544367s FAIL (2025-08-07 21:28:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001564404s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.000831861s FAIL (2025-08-07 21:28:10 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e exceeded Aug 7 21:21:03.692382456 (4h54m17.279438677s) [upload=5064]: Starting upload Aug 7 21:21:03.692428231 (4h54m17.279484453s) [upload=5064]: Done starting upload Aug 7 21:21:03.692467736 (4h54m17.279523957s) [upload=5064]: Uploading segments... Aug 7 21:21:03.692495146 (4h54m17.279551367s) [upload=5064]: Got next segment. Inline: true Aug 7 21:21:03.692500066 (4h54m17.279556287s) [upload=5064]: Waiting for error group managing segments... Aug 7 21:21:03.692529661 (4h54m17.279585882s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.692533137 (4h54m17.279589349s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:03.692535481 (4h54m17.279591702s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:33.679184065 (4h54m47.266240287s) [upload=5064]: Done uploading segments... Aug 7 21:21:33.679200356 (4h54m47.266256568s) [upload=5064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:27.243732093 (4h58m40.830788304s) [upload=5132]: Starting upload Aug 7 21:25:27.243787377 (4h58m40.830843598s) [upload=5132]: Done starting upload Aug 7 21:25:27.243846396 (4h58m40.830902607s) [upload=5132]: Uploading segments... Aug 7 21:25:27.243883505 (4h58m40.830939727s) [upload=5132]: Got next segment. Inline: true Aug 7 21:25:27.243887303 (4h58m40.830943524s) [upload=5132]: Waiting for error group managing segments... Aug 7 21:25:27.243942507 (4h58m40.830998728s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:27.243946624 (4h58m40.831002845s) [upload=5132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:27.243948898 (4h58m40.831005119s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.223887428 (4h59m10.810943669s) [upload=5132]: Done uploading segments... Aug 7 21:25:57.223909498 (4h59m10.81096574s) [upload=5132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:29.142150756 (4h59m42.729206977s) [upload=5147]: Starting upload Aug 7 21:26:29.142194398 (4h59m42.729250619s) [upload=5147]: Done starting upload Aug 7 21:26:29.142233742 (4h59m42.729289963s) [upload=5147]: Uploading segments... Aug 7 21:26:29.142260401 (4h59m42.729316612s) [upload=5147]: Got next segment. Inline: true Aug 7 21:26:29.142263868 (4h59m42.729320089s) [upload=5147]: Waiting for error group managing segments... Aug 7 21:26:29.142293002 (4h59m42.729349213s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.142297080 (4h59m42.729353301s) [upload=5147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:29.142299354 (4h59m42.729355575s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:59.127042625 (5h0m12.714098866s) [upload=5147]: Done uploading segments... Aug 7 21:26:59.127117424 (5h0m12.714173665s) [upload=5147]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:10.814290180 (5h0m24.401346421s) [upload=5160]: Starting upload Aug 7 21:27:10.814371522 (5h0m24.401427764s) [upload=5160]: Done starting upload Aug 7 21:27:10.814438597 (5h0m24.401494839s) [upload=5160]: Uploading segments... Aug 7 21:27:10.814480476 (5h0m24.401536717s) [upload=5160]: Got next segment. Inline: true Aug 7 21:27:10.814485826 (5h0m24.401542067s) [upload=5160]: Waiting for error group managing segments... Aug 7 21:27:10.814529677 (5h0m24.401585918s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:10.814535198 (5h0m24.401591438s) [upload=5160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:10.814538994 (5h0m24.401595236s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:40.714926118 (5h0m54.301982339s) [upload=5160]: Done uploading segments... Aug 7 21:27:40.714942067 (5h0m54.301998278s) [upload=5160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:40.796846333 (5h0m54.383902554s) [upload=5169]: Starting upload Aug 7 21:27:40.796900224 (5h0m54.383956435s) [upload=5169]: Done starting upload Aug 7 21:27:40.796945107 (5h0m54.384001318s) [upload=5169]: Uploading segments... Aug 7 21:27:40.796979782 (5h0m54.384036003s) [upload=5169]: Got next segment. Inline: true Aug 7 21:27:40.796983369 (5h0m54.38403959s) [upload=5169]: Waiting for error group managing segments... Aug 7 21:27:40.797013695 (5h0m54.384069907s) [upload=5169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:40.797017642 (5h0m54.384073863s) [upload=5169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:40.797020007 (5h0m54.384076218s) [upload=5169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:10.771421400 (5h1m24.358477651s) [upload=5169]: Done uploading segments... Aug 7 21:28:10.771479779 (5h1m24.35853602s) [upload=5169]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.0008469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.000733477s FAIL (2025-08-07 21:28:10 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000753945s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.694185331s FAIL (2025-08-07 21:28:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.694210969s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.001232196s FAIL (2025-08-07 21:28:05 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.189934289 (4h54m43.776990511s) [upload=5071]: Starting upload Aug 7 21:21:30.189980476 (4h54m43.777036687s) [upload=5071]: Done starting upload Aug 7 21:21:30.190017424 (4h54m43.777073645s) [upload=5071]: Uploading segments... Aug 7 21:21:30.190043793 (4h54m43.777100005s) [upload=5071]: Got next segment. Inline: true Aug 7 21:21:30.190047100 (4h54m43.777103321s) [upload=5071]: Waiting for error group managing segments... Aug 7 21:21:30.190088458 (4h54m43.777144669s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.190092425 (4h54m43.777148646s) [upload=5071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.190094709 (4h54m43.77715093s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.179283539 (4h55m43.76633979s) [upload=5071]: Done uploading segments... Aug 7 21:22:30.179305229 (4h55m43.766361471s) [upload=5071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.041348848 (4h55m57.628405089s) [upload=5090]: Starting upload Aug 7 21:22:44.041428967 (4h55m57.628485208s) [upload=5090]: Done starting upload Aug 7 21:22:44.041486454 (4h55m57.628542695s) [upload=5090]: Uploading segments... Aug 7 21:22:44.041542930 (4h55m57.628599171s) [upload=5090]: Got next segment. Inline: true Aug 7 21:22:44.041549963 (4h55m57.628606204s) [upload=5090]: Waiting for error group managing segments... Aug 7 21:22:44.041596389 (4h55m57.628652631s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.041602170 (4h55m57.628658411s) [upload=5090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.041605667 (4h55m57.628661917s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.025768658 (4h56m57.612824879s) [upload=5090]: Done uploading segments... Aug 7 21:23:44.025784877 (4h56m57.612841089s) [upload=5090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:58.208173683 (4h57m11.795229924s) [upload=5112]: Starting upload Aug 7 21:23:58.208257669 (4h57m11.795313911s) [upload=5112]: Done starting upload Aug 7 21:23:58.208337199 (4h57m11.79539344s) [upload=5112]: Uploading segments... Aug 7 21:23:58.208378025 (4h57m11.795434265s) [upload=5112]: Got next segment. Inline: true Aug 7 21:23:58.208383675 (4h57m11.795439916s) [upload=5112]: Waiting for error group managing segments... Aug 7 21:23:58.208427437 (4h57m11.795483678s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:58.208433278 (4h57m11.795489519s) [upload=5112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:58.208438388 (4h57m11.795494628s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:58.195766004 (4h58m11.782822225s) [upload=5112]: Done uploading segments... Aug 7 21:24:58.195780731 (4h58m11.782836943s) [upload=5112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:04.854742735 (4h58m18.441798946s) [upload=5127]: Starting upload Aug 7 21:25:04.854807696 (4h58m18.441863907s) [upload=5127]: Done starting upload Aug 7 21:25:04.854870773 (4h58m18.441926984s) [upload=5127]: Uploading segments... Aug 7 21:25:04.854916199 (4h58m18.44197242s) [upload=5127]: Got next segment. Inline: true Aug 7 21:25:04.854936166 (4h58m18.441992387s) [upload=5127]: Waiting for error group managing segments... Aug 7 21:25:04.854972444 (4h58m18.442028655s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:04.854976711 (4h58m18.442032922s) [upload=5127]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:04.854979997 (4h58m18.442036218s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:04.841433540 (4h59m18.428489761s) [upload=5127]: Done uploading segments... Aug 7 21:26:04.841457825 (4h59m18.428514037s) [upload=5127]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:04.958090745 (4h59m18.545146957s) [upload=5139]: Starting upload Aug 7 21:26:04.958138415 (4h59m18.545194626s) [upload=5139]: Done starting upload Aug 7 21:26:04.958184982 (4h59m18.545241203s) [upload=5139]: Uploading segments... Aug 7 21:26:04.958220368 (4h59m18.545276589s) [upload=5139]: Got next segment. Inline: true Aug 7 21:26:04.958224295 (4h59m18.545280516s) [upload=5139]: Waiting for error group managing segments... Aug 7 21:26:04.958253820 (4h59m18.545310031s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:04.958257427 (4h59m18.545313638s) [upload=5139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:04.958259772 (4h59m18.545315993s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:04.939553376 (5h0m18.526609617s) [upload=5139]: Done uploading segments... Aug 7 21:27:04.939574906 (5h0m18.526631147s) [upload=5139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:05.045401846 (5h0m18.632458067s) [upload=5157]: Starting upload Aug 7 21:27:05.045447422 (5h0m18.632503643s) [upload=5157]: Done starting upload Aug 7 21:27:05.045500470 (5h0m18.632556691s) [upload=5157]: Uploading segments... Aug 7 21:27:05.045530256 (5h0m18.632586467s) [upload=5157]: Got next segment. Inline: true Aug 7 21:27:05.045533973 (5h0m18.632590194s) [upload=5157]: Waiting for error group managing segments... Aug 7 21:27:05.045565873 (5h0m18.632622094s) [upload=5157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:05.045571213 (5h0m18.632627434s) [upload=5157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:05.045573487 (5h0m18.632629708s) [upload=5157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:05.013376070 (5h1m18.600432281s) [upload=5157]: Done uploading segments... Aug 7 21:28:05.013391970 (5h1m18.600448181s) [upload=5157]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001243808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 20 out of 60 is failed (threshold=3) [......................................XX..XXXXXXXXXXXXXXXXXX] duration: 30.001084048s FAIL (2025-08-07 21:27:59 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00110605s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [................................................X........XX.] duration: 43.470323ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:27:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 43.543129ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X........XX] duration: 2m0.00085102s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:27:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000870065s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.001090651s FAIL (2025-08-07 21:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001103283s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.001400879s FAIL (2025-08-07 21:27:40 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:03.692382456 (4h54m17.279438677s) [upload=5064]: Starting upload Aug 7 21:21:03.692428231 (4h54m17.279484453s) [upload=5064]: Done starting upload Aug 7 21:21:03.692467736 (4h54m17.279523957s) [upload=5064]: Uploading segments... Aug 7 21:21:03.692495146 (4h54m17.279551367s) [upload=5064]: Got next segment. Inline: true Aug 7 21:21:03.692500066 (4h54m17.279556287s) [upload=5064]: Waiting for error group managing segments... Aug 7 21:21:03.692529661 (4h54m17.279585882s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.692533137 (4h54m17.279589349s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:03.692535481 (4h54m17.279591702s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:33.679184065 (4h54m47.266240287s) [upload=5064]: Done uploading segments... Aug 7 21:21:33.679200356 (4h54m47.266256568s) [upload=5064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:27.243732093 (4h58m40.830788304s) [upload=5132]: Starting upload Aug 7 21:25:27.243787377 (4h58m40.830843598s) [upload=5132]: Done starting upload Aug 7 21:25:27.243846396 (4h58m40.830902607s) [upload=5132]: Uploading segments... Aug 7 21:25:27.243883505 (4h58m40.830939727s) [upload=5132]: Got next segment. Inline: true Aug 7 21:25:27.243887303 (4h58m40.830943524s) [upload=5132]: Waiting for error group managing segments... Aug 7 21:25:27.243942507 (4h58m40.830998728s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:27.243946624 (4h58m40.831002845s) [upload=5132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:27.243948898 (4h58m40.831005119s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.223887428 (4h59m10.810943669s) [upload=5132]: Done uploading segments... Aug 7 21:25:57.223909498 (4h59m10.81096574s) [upload=5132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:29.142150756 (4h59m42.729206977s) [upload=5147]: Starting upload Aug 7 21:26:29.142194398 (4h59m42.729250619s) [upload=5147]: Done starting upload Aug 7 21:26:29.142233742 (4h59m42.729289963s) [upload=5147]: Uploading segments... Aug 7 21:26:29.142260401 (4h59m42.729316612s) [upload=5147]: Got next segment. Inline: true Aug 7 21:26:29.142263868 (4h59m42.729320089s) [upload=5147]: Waiting for error group managing segments... Aug 7 21:26:29.142293002 (4h59m42.729349213s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.142297080 (4h59m42.729353301s) [upload=5147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:29.142299354 (4h59m42.729355575s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:59.127042625 (5h0m12.714098866s) [upload=5147]: Done uploading segments... Aug 7 21:26:59.127117424 (5h0m12.714173665s) [upload=5147]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:10.814290180 (5h0m24.401346421s) [upload=5160]: Starting upload Aug 7 21:27:10.814371522 (5h0m24.401427764s) [upload=5160]: Done starting upload Aug 7 21:27:10.814438597 (5h0m24.401494839s) [upload=5160]: Uploading segments... Aug 7 21:27:10.814480476 (5h0m24.401536717s) [upload=5160]: Got next segment. Inline: true Aug 7 21:27:10.814485826 (5h0m24.401542067s) [upload=5160]: Waiting for error group managing segments... Aug 7 21:27:10.814529677 (5h0m24.401585918s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:10.814535198 (5h0m24.401591438s) [upload=5160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:10.814538994 (5h0m24.401595236s) [upload=5160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:40.714926118 (5h0m54.301982339s) [upload=5160]: Done uploading segments... Aug 7 21:27:40.714942067 (5h0m54.301998278s) [upload=5160]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001421837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.444809255s FAIL (2025-08-07 21:27:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.444831095s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.001002647s FAIL (2025-08-07 21:27:35 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00102067s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.0006237s FAIL (2025-08-07 21:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000643608s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.000710199s FAIL (2025-08-07 21:27:29 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== l: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Aug 7 21:25:15.571225478 (4h58m29.158281699s) [upload=5129]: Starting upload Aug 7 21:25:15.571281894 (4h58m29.158338105s) [upload=5129]: Done starting upload Aug 7 21:25:15.571359468 (4h58m29.158415689s) [upload=5129]: Uploading segments... Aug 7 21:25:15.571564180 (4h58m29.158620401s) [upload=5129]: Got next segment. Inline: false Aug 7 21:25:15.571622810 (4h58m29.158679021s) [upload=5129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.571628479 (4h58m29.1586847s) [upload=5129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:45.547010083 (4h58m59.134066294s) [upload=5129]: Next segment err: context deadline exceeded Aug 7 21:25:45.547084462 (4h58m59.134140683s) [upload=5129]: Done uploading segments... Aug 7 21:25:45.547107044 (4h58m59.134163266s) [upload=5129]: Upload finished. err: context deadline exceeded Aug 7 21:25:45.641521062 (4h58m59.228577283s) [upload=5137]: Starting upload Aug 7 21:25:45.641578831 (4h58m59.228635041s) [upload=5137]: Done starting upload Aug 7 21:25:45.641631238 (4h58m59.228687459s) [upload=5137]: Uploading segments... Aug 7 21:25:45.641794333 (4h58m59.228850554s) [upload=5137]: Got next segment. Inline: false Aug 7 21:25:45.641818608 (4h58m59.228874829s) [upload=5137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.641822736 (4h58m59.228878947s) [upload=5137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:15.630688562 (4h59m29.217744793s) [upload=5137]: Next segment err: context deadline exceeded Aug 7 21:26:15.630873647 (4h59m29.217929858s) [upload=5137]: Done uploading segments... Aug 7 21:26:15.630892703 (4h59m29.217948914s) [upload=5137]: Upload finished. err: context deadline exceeded Aug 7 21:26:29.805240268 (4h59m43.392296479s) [upload=5148]: Starting upload Aug 7 21:26:29.805289190 (4h59m43.392345411s) [upload=5148]: Done starting upload Aug 7 21:26:29.805393004 (4h59m43.392449225s) [upload=5148]: Uploading segments... Aug 7 21:26:29.805516605 (4h59m43.392572816s) [upload=5148]: Got next segment. Inline: false Aug 7 21:26:29.805549756 (4h59m43.392605967s) [upload=5148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.805553633 (4h59m43.392609854s) [upload=5148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:59.790298417 (5h0m13.377354628s) [upload=5148]: Next segment err: context deadline exceeded Aug 7 21:26:59.790450801 (5h0m13.377507012s) [upload=5148]: Done uploading segments... Aug 7 21:26:59.790479374 (5h0m13.377535595s) [upload=5148]: Upload finished. err: context deadline exceeded Aug 7 21:26:59.854535904 (5h0m13.441592145s) [upload=5156]: Starting upload Aug 7 21:26:59.854627225 (5h0m13.441683465s) [upload=5156]: Done starting upload Aug 7 21:26:59.854723153 (5h0m13.441779394s) [upload=5156]: Uploading segments... Aug 7 21:26:59.855007455 (5h0m13.442063705s) [upload=5156]: Got next segment. Inline: false Aug 7 21:26:59.855059541 (5h0m13.442115782s) [upload=5156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:59.855096871 (5h0m13.442153123s) [upload=5156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:29.841757946 (5h0m43.428814187s) [upload=5156]: Next segment err: context deadline exceeded Aug 7 21:27:29.841960274 (5h0m43.429016514s) [upload=5156]: Done uploading segments... Aug 7 21:27:29.842007351 (5h0m43.429063593s) [upload=5156]: Upload finished. err: context deadline exceeded Duration: 30.000739514s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.000626628s FAIL (2025-08-07 21:27:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000642296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 423.421319ms FAIL (2025-08-07 21:27:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 423.449801ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.......................................XX..XXXXXXXXXXXXXXXXX] duration: 30.001559002s FAIL (2025-08-07 21:27:13 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001571466s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.001696662s FAIL (2025-08-07 21:27:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.189934289 (4h54m43.776990511s) [upload=5071]: Starting upload Aug 7 21:21:30.189980476 (4h54m43.777036687s) [upload=5071]: Done starting upload Aug 7 21:21:30.190017424 (4h54m43.777073645s) [upload=5071]: Uploading segments... Aug 7 21:21:30.190043793 (4h54m43.777100005s) [upload=5071]: Got next segment. Inline: true Aug 7 21:21:30.190047100 (4h54m43.777103321s) [upload=5071]: Waiting for error group managing segments... Aug 7 21:21:30.190088458 (4h54m43.777144669s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.190092425 (4h54m43.777148646s) [upload=5071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.190094709 (4h54m43.77715093s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.179283539 (4h55m43.76633979s) [upload=5071]: Done uploading segments... Aug 7 21:22:30.179305229 (4h55m43.766361471s) [upload=5071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.041348848 (4h55m57.628405089s) [upload=5090]: Starting upload Aug 7 21:22:44.041428967 (4h55m57.628485208s) [upload=5090]: Done starting upload Aug 7 21:22:44.041486454 (4h55m57.628542695s) [upload=5090]: Uploading segments... Aug 7 21:22:44.041542930 (4h55m57.628599171s) [upload=5090]: Got next segment. Inline: true Aug 7 21:22:44.041549963 (4h55m57.628606204s) [upload=5090]: Waiting for error group managing segments... Aug 7 21:22:44.041596389 (4h55m57.628652631s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.041602170 (4h55m57.628658411s) [upload=5090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.041605667 (4h55m57.628661917s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.025768658 (4h56m57.612824879s) [upload=5090]: Done uploading segments... Aug 7 21:23:44.025784877 (4h56m57.612841089s) [upload=5090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:58.208173683 (4h57m11.795229924s) [upload=5112]: Starting upload Aug 7 21:23:58.208257669 (4h57m11.795313911s) [upload=5112]: Done starting upload Aug 7 21:23:58.208337199 (4h57m11.79539344s) [upload=5112]: Uploading segments... Aug 7 21:23:58.208378025 (4h57m11.795434265s) [upload=5112]: Got next segment. Inline: true Aug 7 21:23:58.208383675 (4h57m11.795439916s) [upload=5112]: Waiting for error group managing segments... Aug 7 21:23:58.208427437 (4h57m11.795483678s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:58.208433278 (4h57m11.795489519s) [upload=5112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:58.208438388 (4h57m11.795494628s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:58.195766004 (4h58m11.782822225s) [upload=5112]: Done uploading segments... Aug 7 21:24:58.195780731 (4h58m11.782836943s) [upload=5112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:04.854742735 (4h58m18.441798946s) [upload=5127]: Starting upload Aug 7 21:25:04.854807696 (4h58m18.441863907s) [upload=5127]: Done starting upload Aug 7 21:25:04.854870773 (4h58m18.441926984s) [upload=5127]: Uploading segments... Aug 7 21:25:04.854916199 (4h58m18.44197242s) [upload=5127]: Got next segment. Inline: true Aug 7 21:25:04.854936166 (4h58m18.441992387s) [upload=5127]: Waiting for error group managing segments... Aug 7 21:25:04.854972444 (4h58m18.442028655s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:04.854976711 (4h58m18.442032922s) [upload=5127]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:04.854979997 (4h58m18.442036218s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:04.841433540 (4h59m18.428489761s) [upload=5127]: Done uploading segments... Aug 7 21:26:04.841457825 (4h59m18.428514037s) [upload=5127]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:04.958090745 (4h59m18.545146957s) [upload=5139]: Starting upload Aug 7 21:26:04.958138415 (4h59m18.545194626s) [upload=5139]: Done starting upload Aug 7 21:26:04.958184982 (4h59m18.545241203s) [upload=5139]: Uploading segments... Aug 7 21:26:04.958220368 (4h59m18.545276589s) [upload=5139]: Got next segment. Inline: true Aug 7 21:26:04.958224295 (4h59m18.545280516s) [upload=5139]: Waiting for error group managing segments... Aug 7 21:26:04.958253820 (4h59m18.545310031s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:04.958257427 (4h59m18.545313638s) [upload=5139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:04.958259772 (4h59m18.545315993s) [upload=5139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:04.939553376 (5h0m18.526609617s) [upload=5139]: Done uploading segments... Aug 7 21:27:04.939574906 (5h0m18.526631147s) [upload=5139]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001708844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.............................................XX.XXXXXXXXXXXX] duration: 1m0.000601948s FAIL (2025-08-07 21:27:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00061983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.00058487s FAIL (2025-08-07 21:26:59 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Aug 7 21:25:15.571225478 (4h58m29.158281699s) [upload=5129]: Starting upload Aug 7 21:25:15.571281894 (4h58m29.158338105s) [upload=5129]: Done starting upload Aug 7 21:25:15.571359468 (4h58m29.158415689s) [upload=5129]: Uploading segments... Aug 7 21:25:15.571564180 (4h58m29.158620401s) [upload=5129]: Got next segment. Inline: false Aug 7 21:25:15.571622810 (4h58m29.158679021s) [upload=5129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.571628479 (4h58m29.1586847s) [upload=5129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:45.547010083 (4h58m59.134066294s) [upload=5129]: Next segment err: context deadline exceeded Aug 7 21:25:45.547084462 (4h58m59.134140683s) [upload=5129]: Done uploading segments... Aug 7 21:25:45.547107044 (4h58m59.134163266s) [upload=5129]: Upload finished. err: context deadline exceeded Aug 7 21:25:45.641521062 (4h58m59.228577283s) [upload=5137]: Starting upload Aug 7 21:25:45.641578831 (4h58m59.228635041s) [upload=5137]: Done starting upload Aug 7 21:25:45.641631238 (4h58m59.228687459s) [upload=5137]: Uploading segments... Aug 7 21:25:45.641794333 (4h58m59.228850554s) [upload=5137]: Got next segment. Inline: false Aug 7 21:25:45.641818608 (4h58m59.228874829s) [upload=5137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.641822736 (4h58m59.228878947s) [upload=5137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:15.630688562 (4h59m29.217744793s) [upload=5137]: Next segment err: context deadline exceeded Aug 7 21:26:15.630873647 (4h59m29.217929858s) [upload=5137]: Done uploading segments... Aug 7 21:26:15.630892703 (4h59m29.217948914s) [upload=5137]: Upload finished. err: context deadline exceeded Aug 7 21:26:29.805240268 (4h59m43.392296479s) [upload=5148]: Starting upload Aug 7 21:26:29.805289190 (4h59m43.392345411s) [upload=5148]: Done starting upload Aug 7 21:26:29.805393004 (4h59m43.392449225s) [upload=5148]: Uploading segments... Aug 7 21:26:29.805516605 (4h59m43.392572816s) [upload=5148]: Got next segment. Inline: false Aug 7 21:26:29.805549756 (4h59m43.392605967s) [upload=5148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.805553633 (4h59m43.392609854s) [upload=5148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:59.790298417 (5h0m13.377354628s) [upload=5148]: Next segment err: context deadline exceeded Aug 7 21:26:59.790450801 (5h0m13.377507012s) [upload=5148]: Done uploading segments... Aug 7 21:26:59.790479374 (5h0m13.377535595s) [upload=5148]: Upload finished. err: context deadline exceeded Duration: 30.000601181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000813438s FAIL (2025-08-07 21:26:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:03.692382456 (4h54m17.279438677s) [upload=5064]: Starting upload Aug 7 21:21:03.692428231 (4h54m17.279484453s) [upload=5064]: Done starting upload Aug 7 21:21:03.692467736 (4h54m17.279523957s) [upload=5064]: Uploading segments... Aug 7 21:21:03.692495146 (4h54m17.279551367s) [upload=5064]: Got next segment. Inline: true Aug 7 21:21:03.692500066 (4h54m17.279556287s) [upload=5064]: Waiting for error group managing segments... Aug 7 21:21:03.692529661 (4h54m17.279585882s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.692533137 (4h54m17.279589349s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:03.692535481 (4h54m17.279591702s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:33.679184065 (4h54m47.266240287s) [upload=5064]: Done uploading segments... Aug 7 21:21:33.679200356 (4h54m47.266256568s) [upload=5064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:27.243732093 (4h58m40.830788304s) [upload=5132]: Starting upload Aug 7 21:25:27.243787377 (4h58m40.830843598s) [upload=5132]: Done starting upload Aug 7 21:25:27.243846396 (4h58m40.830902607s) [upload=5132]: Uploading segments... Aug 7 21:25:27.243883505 (4h58m40.830939727s) [upload=5132]: Got next segment. Inline: true Aug 7 21:25:27.243887303 (4h58m40.830943524s) [upload=5132]: Waiting for error group managing segments... Aug 7 21:25:27.243942507 (4h58m40.830998728s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:27.243946624 (4h58m40.831002845s) [upload=5132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:27.243948898 (4h58m40.831005119s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.223887428 (4h59m10.810943669s) [upload=5132]: Done uploading segments... Aug 7 21:25:57.223909498 (4h59m10.81096574s) [upload=5132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:29.142150756 (4h59m42.729206977s) [upload=5147]: Starting upload Aug 7 21:26:29.142194398 (4h59m42.729250619s) [upload=5147]: Done starting upload Aug 7 21:26:29.142233742 (4h59m42.729289963s) [upload=5147]: Uploading segments... Aug 7 21:26:29.142260401 (4h59m42.729316612s) [upload=5147]: Got next segment. Inline: true Aug 7 21:26:29.142263868 (4h59m42.729320089s) [upload=5147]: Waiting for error group managing segments... Aug 7 21:26:29.142293002 (4h59m42.729349213s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:29.142297080 (4h59m42.729353301s) [upload=5147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:29.142299354 (4h59m42.729355575s) [upload=5147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:59.127042625 (5h0m12.714098866s) [upload=5147]: Done uploading segments... Aug 7 21:26:59.127117424 (5h0m12.714173665s) [upload=5147]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000833386s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.000735278s FAIL (2025-08-07 21:26:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000747651s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 18 out of 60 is failed (threshold=3) [........................................XX..XXXXXXXXXXXXXXXX] duration: 30.001059687s FAIL (2025-08-07 21:26:43 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001073913s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.000374749s FAIL (2025-08-07 21:26:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000398043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.000646065s FAIL (2025-08-07 21:26:35 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000659149s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.163222713s FAIL (2025-08-07 21:26:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.163244132s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.001906959s FAIL (2025-08-07 21:26:15 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== V3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532867749 (4h52m8.11992399s) [seg_pos={0 0}, upload=5025]: Segment reached optimal threshold of 13 pieces. Aug 7 21:18:54.533278306 (4h52m8.120334548s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533410502 (4h52m8.120466743s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Aug 7 21:25:15.571225478 (4h58m29.158281699s) [upload=5129]: Starting upload Aug 7 21:25:15.571281894 (4h58m29.158338105s) [upload=5129]: Done starting upload Aug 7 21:25:15.571359468 (4h58m29.158415689s) [upload=5129]: Uploading segments... Aug 7 21:25:15.571564180 (4h58m29.158620401s) [upload=5129]: Got next segment. Inline: false Aug 7 21:25:15.571622810 (4h58m29.158679021s) [upload=5129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.571628479 (4h58m29.1586847s) [upload=5129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:45.547010083 (4h58m59.134066294s) [upload=5129]: Next segment err: context deadline exceeded Aug 7 21:25:45.547084462 (4h58m59.134140683s) [upload=5129]: Done uploading segments... Aug 7 21:25:45.547107044 (4h58m59.134163266s) [upload=5129]: Upload finished. err: context deadline exceeded Aug 7 21:25:45.641521062 (4h58m59.228577283s) [upload=5137]: Starting upload Aug 7 21:25:45.641578831 (4h58m59.228635041s) [upload=5137]: Done starting upload Aug 7 21:25:45.641631238 (4h58m59.228687459s) [upload=5137]: Uploading segments... Aug 7 21:25:45.641794333 (4h58m59.228850554s) [upload=5137]: Got next segment. Inline: false Aug 7 21:25:45.641818608 (4h58m59.228874829s) [upload=5137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.641822736 (4h58m59.228878947s) [upload=5137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:15.630688562 (4h59m29.217744793s) [upload=5137]: Next segment err: context deadline exceeded Aug 7 21:26:15.630873647 (4h59m29.217929858s) [upload=5137]: Done uploading segments... Aug 7 21:26:15.630892703 (4h59m29.217948914s) [upload=5137]: Upload finished. err: context deadline exceeded Duration: 30.001919362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.........................................XX..XXXXXXXXXXXXXXX] duration: 30.000864754s FAIL (2025-08-07 21:26:13 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000939393s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.000966493s FAIL (2025-08-07 21:26:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.189934289 (4h54m43.776990511s) [upload=5071]: Starting upload Aug 7 21:21:30.189980476 (4h54m43.777036687s) [upload=5071]: Done starting upload Aug 7 21:21:30.190017424 (4h54m43.777073645s) [upload=5071]: Uploading segments... Aug 7 21:21:30.190043793 (4h54m43.777100005s) [upload=5071]: Got next segment. Inline: true Aug 7 21:21:30.190047100 (4h54m43.777103321s) [upload=5071]: Waiting for error group managing segments... Aug 7 21:21:30.190088458 (4h54m43.777144669s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.190092425 (4h54m43.777148646s) [upload=5071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.190094709 (4h54m43.77715093s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.179283539 (4h55m43.76633979s) [upload=5071]: Done uploading segments... Aug 7 21:22:30.179305229 (4h55m43.766361471s) [upload=5071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.041348848 (4h55m57.628405089s) [upload=5090]: Starting upload Aug 7 21:22:44.041428967 (4h55m57.628485208s) [upload=5090]: Done starting upload Aug 7 21:22:44.041486454 (4h55m57.628542695s) [upload=5090]: Uploading segments... Aug 7 21:22:44.041542930 (4h55m57.628599171s) [upload=5090]: Got next segment. Inline: true Aug 7 21:22:44.041549963 (4h55m57.628606204s) [upload=5090]: Waiting for error group managing segments... Aug 7 21:22:44.041596389 (4h55m57.628652631s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.041602170 (4h55m57.628658411s) [upload=5090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.041605667 (4h55m57.628661917s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.025768658 (4h56m57.612824879s) [upload=5090]: Done uploading segments... Aug 7 21:23:44.025784877 (4h56m57.612841089s) [upload=5090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:58.208173683 (4h57m11.795229924s) [upload=5112]: Starting upload Aug 7 21:23:58.208257669 (4h57m11.795313911s) [upload=5112]: Done starting upload Aug 7 21:23:58.208337199 (4h57m11.79539344s) [upload=5112]: Uploading segments... Aug 7 21:23:58.208378025 (4h57m11.795434265s) [upload=5112]: Got next segment. Inline: true Aug 7 21:23:58.208383675 (4h57m11.795439916s) [upload=5112]: Waiting for error group managing segments... Aug 7 21:23:58.208427437 (4h57m11.795483678s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:58.208433278 (4h57m11.795489519s) [upload=5112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:58.208438388 (4h57m11.795494628s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:58.195766004 (4h58m11.782822225s) [upload=5112]: Done uploading segments... Aug 7 21:24:58.195780731 (4h58m11.782836943s) [upload=5112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:04.854742735 (4h58m18.441798946s) [upload=5127]: Starting upload Aug 7 21:25:04.854807696 (4h58m18.441863907s) [upload=5127]: Done starting upload Aug 7 21:25:04.854870773 (4h58m18.441926984s) [upload=5127]: Uploading segments... Aug 7 21:25:04.854916199 (4h58m18.44197242s) [upload=5127]: Got next segment. Inline: true Aug 7 21:25:04.854936166 (4h58m18.441992387s) [upload=5127]: Waiting for error group managing segments... Aug 7 21:25:04.854972444 (4h58m18.442028655s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:04.854976711 (4h58m18.442032922s) [upload=5127]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:04.854979997 (4h58m18.442036218s) [upload=5127]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:04.841433540 (4h59m18.428489761s) [upload=5127]: Done uploading segments... Aug 7 21:26:04.841457825 (4h59m18.428514037s) [upload=5127]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000982633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.00128825s FAIL (2025-08-07 21:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001301804s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..............................................XX.XXXXXXXXXXX] duration: 1m0.000799585s FAIL (2025-08-07 21:25:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000812079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.000676549s FAIL (2025-08-07 21:25:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:03.692382456 (4h54m17.279438677s) [upload=5064]: Starting upload Aug 7 21:21:03.692428231 (4h54m17.279484453s) [upload=5064]: Done starting upload Aug 7 21:21:03.692467736 (4h54m17.279523957s) [upload=5064]: Uploading segments... Aug 7 21:21:03.692495146 (4h54m17.279551367s) [upload=5064]: Got next segment. Inline: true Aug 7 21:21:03.692500066 (4h54m17.279556287s) [upload=5064]: Waiting for error group managing segments... Aug 7 21:21:03.692529661 (4h54m17.279585882s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.692533137 (4h54m17.279589349s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:03.692535481 (4h54m17.279591702s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:33.679184065 (4h54m47.266240287s) [upload=5064]: Done uploading segments... Aug 7 21:21:33.679200356 (4h54m47.266256568s) [upload=5064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:27.243732093 (4h58m40.830788304s) [upload=5132]: Starting upload Aug 7 21:25:27.243787377 (4h58m40.830843598s) [upload=5132]: Done starting upload Aug 7 21:25:27.243846396 (4h58m40.830902607s) [upload=5132]: Uploading segments... Aug 7 21:25:27.243883505 (4h58m40.830939727s) [upload=5132]: Got next segment. Inline: true Aug 7 21:25:27.243887303 (4h58m40.830943524s) [upload=5132]: Waiting for error group managing segments... Aug 7 21:25:27.243942507 (4h58m40.830998728s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:27.243946624 (4h58m40.831002845s) [upload=5132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:27.243948898 (4h58m40.831005119s) [upload=5132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.223887428 (4h59m10.810943669s) [upload=5132]: Done uploading segments... Aug 7 21:25:57.223909498 (4h59m10.81096574s) [upload=5132]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000694893s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 2m0.001173896s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:25:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.00119682s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.001434484s FAIL (2025-08-07 21:25:45 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.284511737 (4h52m7.871567998s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.291770877 (4h52m7.878827127s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.298214294 (4h52m7.885270545s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532849875 (4h52m8.119906117s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532867749 (4h52m8.11992399s) [seg_pos={0 0}, upload=5025]: Segment reached optimal threshold of 13 pieces. Aug 7 21:18:54.533278306 (4h52m8.120334548s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533410502 (4h52m8.120466743s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Aug 7 21:25:15.571225478 (4h58m29.158281699s) [upload=5129]: Starting upload Aug 7 21:25:15.571281894 (4h58m29.158338105s) [upload=5129]: Done starting upload Aug 7 21:25:15.571359468 (4h58m29.158415689s) [upload=5129]: Uploading segments... Aug 7 21:25:15.571564180 (4h58m29.158620401s) [upload=5129]: Got next segment. Inline: false Aug 7 21:25:15.571622810 (4h58m29.158679021s) [upload=5129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.571628479 (4h58m29.1586847s) [upload=5129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:45.547010083 (4h58m59.134066294s) [upload=5129]: Next segment err: context deadline exceeded Aug 7 21:25:45.547084462 (4h58m59.134140683s) [upload=5129]: Done uploading segments... Aug 7 21:25:45.547107044 (4h58m59.134163266s) [upload=5129]: Upload finished. err: context deadline exceeded Duration: 30.001446367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.001417385s FAIL (2025-08-07 21:25:33 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001434938s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.000698644s FAIL (2025-08-07 21:25:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000717209s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.000360792s FAIL (2025-08-07 21:25:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000376241s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.000497377s FAIL (2025-08-07 21:25:26 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:03.692382456 (4h54m17.279438677s) [upload=5064]: Starting upload Aug 7 21:21:03.692428231 (4h54m17.279484453s) [upload=5064]: Done starting upload Aug 7 21:21:03.692467736 (4h54m17.279523957s) [upload=5064]: Uploading segments... Aug 7 21:21:03.692495146 (4h54m17.279551367s) [upload=5064]: Got next segment. Inline: true Aug 7 21:21:03.692500066 (4h54m17.279556287s) [upload=5064]: Waiting for error group managing segments... Aug 7 21:21:03.692529661 (4h54m17.279585882s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.692533137 (4h54m17.279589349s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:03.692535481 (4h54m17.279591702s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:33.679184065 (4h54m47.266240287s) [upload=5064]: Done uploading segments... Aug 7 21:21:33.679200356 (4h54m47.266256568s) [upload=5064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:41.028727083 (4h54m54.615783293s) [upload=5077]: Starting upload Aug 7 21:21:41.028776044 (4h54m54.615832265s) [upload=5077]: Done starting upload Aug 7 21:21:41.028822871 (4h54m54.615879093s) [upload=5077]: Uploading segments... Aug 7 21:21:41.028849481 (4h54m54.615905693s) [upload=5077]: Got next segment. Inline: true Aug 7 21:21:41.028853158 (4h54m54.61590938s) [upload=5077]: Waiting for error group managing segments... Aug 7 21:21:41.028887172 (4h54m54.615943383s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:41.028891160 (4h54m54.615947381s) [upload=5077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:41.028893564 (4h54m54.615949785s) [upload=5077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:11.020606617 (4h55m24.607662858s) [upload=5077]: Done uploading segments... Aug 7 21:22:11.020629420 (4h55m24.60768566s) [upload=5077]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.704346518 (4h55m58.291402739s) [upload=5092]: Starting upload Aug 7 21:22:44.704395569 (4h55m58.29145179s) [upload=5092]: Done starting upload Aug 7 21:22:44.704432468 (4h55m58.291488689s) [upload=5092]: Uploading segments... Aug 7 21:22:44.704457685 (4h55m58.291513906s) [upload=5092]: Got next segment. Inline: true Aug 7 21:22:44.704461182 (4h55m58.291517403s) [upload=5092]: Waiting for error group managing segments... Aug 7 21:22:44.704487161 (4h55m58.291543382s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.704490647 (4h55m58.291546868s) [upload=5092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.704493612 (4h55m58.291549833s) [upload=5092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.694775680 (4h56m28.281831901s) [upload=5092]: Done uploading segments... Aug 7 21:23:14.694790848 (4h56m28.281847069s) [upload=5092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.781991861 (4h56m28.369048072s) [upload=5099]: Starting upload Aug 7 21:23:14.782035392 (4h56m28.369091604s) [upload=5099]: Done starting upload Aug 7 21:23:14.782087559 (4h56m28.369143771s) [upload=5099]: Uploading segments... Aug 7 21:23:14.782124759 (4h56m28.369180991s) [upload=5099]: Got next segment. Inline: true Aug 7 21:23:14.782130029 (4h56m28.369186241s) [upload=5099]: Waiting for error group managing segments... Aug 7 21:23:14.782161368 (4h56m28.369217579s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.782164914 (4h56m28.369221126s) [upload=5099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.782167850 (4h56m28.369224061s) [upload=5099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.771505754 (4h56m58.358561975s) [upload=5099]: Done uploading segments... Aug 7 21:23:44.771536371 (4h56m58.358592583s) [upload=5099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:44.852200021 (4h56m58.439256242s) [upload=5107]: Starting upload Aug 7 21:23:44.852251448 (4h56m58.439307669s) [upload=5107]: Done starting upload Aug 7 21:23:44.852291703 (4h56m58.439347924s) [upload=5107]: Uploading segments... Aug 7 21:23:44.852320757 (4h56m58.439376978s) [upload=5107]: Got next segment. Inline: true Aug 7 21:23:44.852324224 (4h56m58.439380445s) [upload=5107]: Waiting for error group managing segments... Aug 7 21:23:44.852355542 (4h56m58.439411753s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:44.852359279 (4h56m58.4394155s) [upload=5107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:44.852361614 (4h56m58.439417824s) [upload=5107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.835050939 (4h57m28.42210715s) [upload=5107]: Done uploading segments... Aug 7 21:24:14.835115480 (4h57m28.422171701s) [upload=5107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:56.715637130 (4h58m10.302693351s) [upload=5124]: Starting upload Aug 7 21:24:56.715691522 (4h58m10.302747743s) [upload=5124]: Done starting upload Aug 7 21:24:56.715738660 (4h58m10.302794881s) [upload=5124]: Uploading segments... Aug 7 21:24:56.715821674 (4h58m10.302877895s) [upload=5124]: Got next segment. Inline: true Aug 7 21:24:56.715825752 (4h58m10.302881973s) [upload=5124]: Waiting for error group managing segments... Aug 7 21:24:56.715858083 (4h58m10.302914294s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:56.715861810 (4h58m10.302918031s) [upload=5124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:56.715864014 (4h58m10.302920235s) [upload=5124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:26.698879559 (4h58m40.285935771s) [upload=5124]: Done uploading segments... Aug 7 21:25:26.698896120 (4h58m40.285952331s) [upload=5124]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000514088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.000570373s FAIL (2025-08-07 21:25:15 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== C7XHQA37JIKXDA7SSO6HOVCCGDS2SHWP7ROPG6MHQ7ZA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.272541713 (4h52m7.859597954s) [noise=true , address=45.91.116.14:4189 , piece_id=GRDSWIHHZ4FEWBEHRY7WPEF546PJWGBIXXCYDESVV4N3GSAWVGYA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.273006260 (4h52m7.860062501s) [noise=true , address=45.91.116.8:4047 , piece_id=7F4MB4X3CJZH3DUXC7VXN3HGQBBRFUV62G4KLCKV7BSARABJENWQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.275575946 (4h52m7.862632188s) [noise=true , address=77.90.31.10:4520 , piece_id=Z5P2H3YFJDC5WMKMFIFVYGFVAU2R27KTNMIAYBDYZGRUT5TWN5QQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.281367938 (4h52m7.868424169s) [noise=true , address=77.90.31.15:4697 , piece_id=PILNZYYWJABT57ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.284511737 (4h52m7.871567998s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.291770877 (4h52m7.878827127s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.298214294 (4h52m7.885270545s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532849875 (4h52m8.119906117s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532867749 (4h52m8.11992399s) [seg_pos={0 0}, upload=5025]: Segment reached optimal threshold of 13 pieces. Aug 7 21:18:54.533278306 (4h52m8.120334548s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533410502 (4h52m8.120466743s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Aug 7 21:21:31.936040213 (4h54m45.523096464s) [upload=5073]: Starting upload Aug 7 21:21:31.936133387 (4h54m45.523189638s) [upload=5073]: Done starting upload Aug 7 21:21:31.936256627 (4h54m45.523312868s) [upload=5073]: Uploading segments... Aug 7 21:21:31.936495583 (4h54m45.523551824s) [upload=5073]: Got next segment. Inline: false Aug 7 21:21:31.936532813 (4h54m45.523589053s) [upload=5073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:31.936537992 (4h54m45.523594234s) [upload=5073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:01.921915749 (4h55m15.50897196s) [upload=5073]: Next segment err: context deadline exceeded Aug 7 21:22:01.922047263 (4h55m15.509103475s) [upload=5073]: Done uploading segments... Aug 7 21:22:01.922079755 (4h55m15.509135976s) [upload=5073]: Upload finished. err: context deadline exceeded Aug 7 21:22:02.000538540 (4h55m15.587594761s) [upload=5082]: Starting upload Aug 7 21:22:02.000624420 (4h55m15.587680652s) [upload=5082]: Done starting upload Aug 7 21:22:02.000735858 (4h55m15.587792089s) [upload=5082]: Uploading segments... Aug 7 21:22:02.000881511 (4h55m15.587937761s) [upload=5082]: Got next segment. Inline: false Aug 7 21:22:02.000947453 (4h55m15.588003684s) [upload=5082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:02.000954897 (4h55m15.588011118s) [upload=5082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:31.979343661 (4h55m45.566399882s) [upload=5082]: Next segment err: context deadline exceeded Aug 7 21:22:31.979438208 (4h55m45.566494418s) [upload=5082]: Done uploading segments... Aug 7 21:22:31.979474655 (4h55m45.566530877s) [upload=5082]: Upload finished. err: context deadline exceeded Aug 7 21:22:41.968187982 (4h55m55.555244194s) [upload=5089]: Starting upload Aug 7 21:22:41.968245641 (4h55m55.555301862s) [upload=5089]: Done starting upload Aug 7 21:22:41.968339846 (4h55m55.555396087s) [upload=5089]: Uploading segments... Aug 7 21:22:41.968596806 (4h55m55.555653047s) [upload=5089]: Got next segment. Inline: false Aug 7 21:22:41.968638122 (4h55m55.555694354s) [upload=5089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:41.968644665 (4h55m55.555700906s) [upload=5089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:11.943981705 (4h56m25.531037956s) [upload=5089]: Next segment err: context deadline exceeded Aug 7 21:23:11.944159476 (4h56m25.531215718s) [upload=5089]: Done uploading segments... Aug 7 21:23:11.944182590 (4h56m25.53123883s) [upload=5089]: Upload finished. err: context deadline exceeded Aug 7 21:23:20.066472018 (4h56m33.653528259s) [upload=5100]: Starting upload Aug 7 21:23:20.066538723 (4h56m33.653594965s) [upload=5100]: Done starting upload Aug 7 21:23:20.066650672 (4h56m33.653706913s) [upload=5100]: Uploading segments... Aug 7 21:23:20.066882696 (4h56m33.653938936s) [upload=5100]: Got next segment. Inline: false Aug 7 21:23:20.066929522 (4h56m33.653985764s) [upload=5100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:20.066934632 (4h56m33.653990873s) [upload=5100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:50.045530632 (4h57m3.632586874s) [upload=5100]: Next segment err: context deadline exceeded Aug 7 21:23:50.045695360 (4h57m3.6327516s) [upload=5100]: Done uploading segments... Aug 7 21:23:50.045720867 (4h57m3.632777109s) [upload=5100]: Upload finished. err: context deadline exceeded Aug 7 21:23:50.132503139 (4h57m3.719559361s) [upload=5111]: Starting upload Aug 7 21:23:50.132542101 (4h57m3.719598322s) [upload=5111]: Done starting upload Aug 7 21:23:50.132574512 (4h57m3.719630733s) [upload=5111]: Uploading segments... Aug 7 21:23:50.132772312 (4h57m3.719828533s) [upload=5111]: Got next segment. Inline: false Aug 7 21:23:50.132810694 (4h57m3.719866904s) [upload=5111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.132814390 (4h57m3.719870611s) [upload=5111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:20.119434924 (4h57m33.706491135s) [upload=5111]: Next segment err: context deadline exceeded Aug 7 21:24:20.119485478 (4h57m33.706541699s) [upload=5111]: Done uploading segments... Aug 7 21:24:20.119497891 (4h57m33.706554102s) [upload=5111]: Upload finished. err: context deadline exceeded Aug 7 21:24:45.494243280 (4h57m59.081299501s) [upload=5120]: Starting upload Aug 7 21:24:45.494313381 (4h57m59.081369612s) [upload=5120]: Done starting upload Aug 7 21:24:45.494474943 (4h57m59.081531184s) [upload=5120]: Uploading segments... Aug 7 21:24:45.494627989 (4h57m59.08168422s) [upload=5120]: Got next segment. Inline: false Aug 7 21:24:45.494663625 (4h57m59.081719856s) [upload=5120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:45.494668073 (4h57m59.081724294s) [upload=5120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:15.480538548 (4h58m29.067594769s) [upload=5120]: Next segment err: context deadline exceeded Aug 7 21:25:15.480665896 (4h58m29.067722117s) [upload=5120]: Done uploading segments... Aug 7 21:25:15.480717081 (4h58m29.067773302s) [upload=5120]: Upload finished. err: context deadline exceeded Duration: 30.000592735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 548.137409ms FAIL (2025-08-07 21:25:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 548.188994ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [..........................................XX..XXXXXXXXXXXXXX] duration: 30.000991155s FAIL (2025-08-07 21:25:03 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001027953s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.000430116s FAIL (2025-08-07 21:24:58 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.189934289 (4h54m43.776990511s) [upload=5071]: Starting upload Aug 7 21:21:30.189980476 (4h54m43.777036687s) [upload=5071]: Done starting upload Aug 7 21:21:30.190017424 (4h54m43.777073645s) [upload=5071]: Uploading segments... Aug 7 21:21:30.190043793 (4h54m43.777100005s) [upload=5071]: Got next segment. Inline: true Aug 7 21:21:30.190047100 (4h54m43.777103321s) [upload=5071]: Waiting for error group managing segments... Aug 7 21:21:30.190088458 (4h54m43.777144669s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.190092425 (4h54m43.777148646s) [upload=5071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.190094709 (4h54m43.77715093s) [upload=5071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.179283539 (4h55m43.76633979s) [upload=5071]: Done uploading segments... Aug 7 21:22:30.179305229 (4h55m43.766361471s) [upload=5071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:44.041348848 (4h55m57.628405089s) [upload=5090]: Starting upload Aug 7 21:22:44.041428967 (4h55m57.628485208s) [upload=5090]: Done starting upload Aug 7 21:22:44.041486454 (4h55m57.628542695s) [upload=5090]: Uploading segments... Aug 7 21:22:44.041542930 (4h55m57.628599171s) [upload=5090]: Got next segment. Inline: true Aug 7 21:22:44.041549963 (4h55m57.628606204s) [upload=5090]: Waiting for error group managing segments... Aug 7 21:22:44.041596389 (4h55m57.628652631s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:44.041602170 (4h55m57.628658411s) [upload=5090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:44.041605667 (4h55m57.628661917s) [upload=5090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:44.025768658 (4h56m57.612824879s) [upload=5090]: Done uploading segments... Aug 7 21:23:44.025784877 (4h56m57.612841089s) [upload=5090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:58.208173683 (4h57m11.795229924s) [upload=5112]: Starting upload Aug 7 21:23:58.208257669 (4h57m11.795313911s) [upload=5112]: Done starting upload Aug 7 21:23:58.208337199 (4h57m11.79539344s) [upload=5112]: Uploading segments... Aug 7 21:23:58.208378025 (4h57m11.795434265s) [upload=5112]: Got next segment. Inline: true Aug 7 21:23:58.208383675 (4h57m11.795439916s) [upload=5112]: Waiting for error group managing segments... Aug 7 21:23:58.208427437 (4h57m11.795483678s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:58.208433278 (4h57m11.795489519s) [upload=5112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:58.208438388 (4h57m11.795494628s) [upload=5112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:58.195766004 (4h58m11.782822225s) [upload=5112]: Done uploading segments... Aug 7 21:24:58.195780731 (4h58m11.782836943s) [upload=5112]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000450373s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)