Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.285492005s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:35:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.285511471s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.867388121s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:34:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.867410804s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.185852763s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.185865897s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.66408232s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:33:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.664102167s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.832115369s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:33:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.832137591s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.645686093s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:33:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.645705129s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.542421033s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:33:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.542442202s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.006445453s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:33:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.006469449s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.920930399s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:32:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.92094779s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.767299301s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:32:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.767321363s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.685831227s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:32:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.685852316s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.59915854s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:31:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.599179038s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.598890287s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:31:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.598914743s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.66164663s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:31:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.661670304s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.675629122s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:30:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.675647897s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.886343019s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:30:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.886360301s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m23.125563648s trace: https://jaeger.dev.storj.io/trace/69e0358420660331 FAIL (2025-12-15 18:30:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.459852640 (243h0m1.157420946s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460031273 (243h0m1.157599589s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460182977 (243h0m1.157751282s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460428956 (243h0m1.157997272s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.460765174 (243h0m1.15833348s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461076576 (243h0m1.158644892s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.461642794 (243h0m1.15921111s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462786149 (243h0m1.160354456s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.462862161 (243h0m1.160430477s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.463249855 (243h0m1.160818171s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Uploading piece... Dec 15 18:29:13.921820969 (243h0m1.619389305s) [upload=213590]: Next returned nil segment Dec 15 18:29:13.921835546 (243h0m1.619403882s) [upload=213590]: Waiting for error group managing segments... Dec 15 18:29:14.102209961 (243h0m1.799778297s) [noise=true , address=185.167.204.206:4041 , piece_id=JUCFJ7ECIFKWL3L5YFHQW3B5BW3LPWFSKJHQEL7R7U3UWLWIGCLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103304245 (243h0m1.800872581s) [noise=true , address=81.171.14.70:4035 , piece_id=GXWOGJ5PECLWO22Y47L5EFVWCHQQDL3T446O6CEPL4BWXEXDYZ6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.103991349 (243h0m1.801559684s) [noise=true , address=185.167.204.205:4031 , piece_id=MEX7UHVCMDZ4S5Z74ZWVO4MGSXAHF5W4WO2QDOWWTY5JJ72BGV6A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105066186 (243h0m1.802634523s) [noise=true , address=185.167.204.206:4059 , piece_id=7XUPDV4PMJK27J6WPZSE6ZG53TNSEVOMKKVOKMRQK5W333A7BUYQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105557644 (243h0m1.80312598s) [noise=true , address=185.167.204.206:4051 , piece_id=2WOO3SS36XVBB3NJI2YEYGA2AKRCGJNJIA2XBHANBBGYDZ54T2NQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.105945328 (243h0m1.803513664s) [noise=true , address=185.167.204.206:4038 , piece_id=NOGNEAEQJ2LD4GRQUSUBGTPBDP6SYPRJ7YTMEXVVBXEYDYQKMPDA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106594641 (243h0m1.804162978s) [noise=true , address=185.167.204.205:4030 , piece_id=ZGYHCOYSCSWT2QZLJCTU3KMO6LPLC7OKATECVQCVH7RVDGDJF5EQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.106626059 (243h0m1.804194396s) [noise=true , address=185.167.204.206:4048 , piece_id=OAYEAM4T72PSS4KUDQ3DLLVTP5MRNIYVPQPMJNOMDSBUYWDYEASA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.111587317 (243h0m1.809155664s) [noise=true , address=185.167.204.205:4006 , piece_id=LEAQGBFOGEOE23XECG53EJZLVD2PWV6CSUS4ARBJODJONRVUZMMA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.113881994 (243h0m1.81145033s) [noise=true , address=185.167.204.205:4004 , piece_id=5MHLK2CHAA7Y63VIGKP3EC2ELIDOFIHEJ7SCD3A2KSOM2ULOJGRQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.114057433 (243h0m1.811625778s) [noise=true , address=185.167.204.205:4014 , piece_id=LY26SU5P3B432UOQOTBPBJU52YF3CP4BOLFKY6FO53JJPUGBC4HA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.135505468 (243h0m1.833073804s) [noise=true , address=185.167.204.205:4015 , piece_id=CZ7JINWLX74OMXCTZ64NMATEGSOD3FXCMVCIY3DDMVL4SDQPITLQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.136080412 (243h0m1.833648758s) [noise=true , address=185.167.204.205:4020 , piece_id=QWXLNCKGCLMKQEC72VWVRDDF7GICOUVMA6HVWH26KCEGFGBO7YQA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.140861924 (243h0m1.83843027s) [noise=true , address=185.167.204.206:4057 , piece_id=Y742D5IJUNUZHM4M5573EJBUTVIF6E47POFKMX53UFNPFSWKBAPA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.144172197 (243h0m1.841740543s) [noise=true , address=185.167.204.205:4017 , piece_id=BV6DKYAOKGWIUJZ3LML4O6RH3EWWG4SIP2OTNNYKXLSBSCIV7KNQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.146741867 (243h0m1.844310204s) [noise=true , address=185.167.204.206:4064 , piece_id=OWPKXUWDAOKVEDKXM5VTLMRKMWR5CYCSIB623LFFOTH6O6YTWQ4A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.149631956 (243h0m1.847200293s) [noise=true , address=81.171.14.70:4039 , piece_id=YXDUZEKD34HEOW3DITW5R4DJRMBJXTZLQQKJQF6XB6EXA7J7R2OQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.150389200 (243h0m1.847957537s) [noise=true , address=89.149.201.147:4009 , piece_id=3H2AI33X3HXAZOWPRO4RWPD3O6PIWG7FP62JMOHEJDBDHINHT2BQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.153533434 (243h0m1.851101771s) [noise=true , address=185.167.204.206:4037 , piece_id=DKDBJBEHXCAUV2NP5APGI2AHE5J2TP5RGLQUAGOTMRF3R6I7WDAQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.155880729 (243h0m1.853449065s) [noise=true , address=185.167.204.206:4067 , piece_id=ZLCNDNR57WPIPI6MGYHC4VDO6HT6ITH67WF34G2SOBNBAYOQ7AUA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.156251001 (243h0m1.853819337s) [noise=true , address=89.149.201.147:4017 , piece_id=HR6F7NN36BWOLUOZJNWSJMRK7CBCRNFAI5K3SJNIFTS4DZFU5COA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.162011121 (243h0m1.859579457s) [noise=true , address=89.149.201.147:4024 , piece_id=75C23RDXZY2PHKV6H5OMEIUN6FA4G6BJIYQXDTQTITANCBI64XSQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.165282262 (243h0m1.862850598s) [noise=true , address=185.167.204.205:4034 , piece_id=UBBPTZUNSMPWAXP6APIEYNLVHOPD7V6B6O6J7SUR4G2UQH2WEZFA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.170533510 (243h0m1.868101847s) [noise=true , address=81.171.14.70:4040 , piece_id=DZXFOHUXZY5XEJZEOAX26D4SFHE24TXBGOENUMCXJVV5KC35HSIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.613159689 (243h0m2.310728024s) [noise=true , address=89.149.201.147:4001 , piece_id=UQ7WHXOFMYDIHHKAS7HKCGJ4DPKJQW5ATS2DJS4AHKKEYCSVOLIQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623817647 (243h0m2.321385983s) [noise=true , address=81.171.14.70:4030 , piece_id=FSUEDURKIR6SQXZ6ARAKUK5NUR7HVZAUYW4UIVOC3KT4U4TGXPZA, seg_pos={0 0}, upload=213590]: Done uploading piece. err: <nil> Dec 15 18:29:14.623840790 (243h0m2.321409127s) [seg_pos={0 0}, upload=213590]: Segment reached optimal threshold of 26 pieces. Dec 15 18:29:14.624285762 (243h0m2.321854098s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624301081 (243h0m2.321869427s) [noise=true , address=81.171.14.70:4028 , piece_id=6N6PZWXGV6ZH6EO5LT2N3XOWZORKYBLLNH6LR5WOLAAYEWDDMI7A, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624824528 (243h0m2.322392864s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 15 18:29:14.624839166 (243h0m2.322407502s) [noise=true , address=89.149.201.147:4012 , piece_id=B4RZ3EB4QK2KHUKNZYVFRSLNSQMTGYBOOTSO7BFV26XMMXYE5RGQ, seg_pos={0 0}, upload=213590]: Long tail cancellation due to optimal threshold Dec 15 18:29:14.624877788 (243h0m2.322446124s) [seg_pos={0 0}, upload=213590]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 15 18:29:14.624977574 (243h0m2.32254591s) [upload=213590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 18:29:14.624988475 (243h0m2.322556811s) [upload=213590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 18:30:36.176310366 (243h1m23.873878673s) [upload=213590]: Done uploading segments... Dec 15 18:30:36.176388972 (243h1m23.873957288s) [upload=213590]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m23.125572295s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 57.119303812s trace: https://jaeger.dev.storj.io/trace/373dcd1aa45f200e FAIL (2025-12-15 18:30:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 57.11932374s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 377.803923ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 377.822258ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 595.733304ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:54:44 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 595.758101ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 599.293162ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:54:00 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 599.313389ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 415.386526ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:53:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.404239ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 410.049404ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.065644ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 549.084027ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:53:21 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 549.098606ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 369.77601ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 369.792431ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 631.971777ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:52:25 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 631.98869ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 425.157108ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:51:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.175483ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration: 581.72892ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:51:31 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 581.748476ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XXXX.X......................................................] duration: 620.143441ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:50:55 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 620.16489ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 410.665834ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.682245ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XXXXX.X.....................................................] duration: 625.066447ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:50:17 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 625.085724ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 398.889728ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.912089ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXX.X....................................................] duration: 552.191561ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:49:19 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 552.209986ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 411.210541ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:48:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.222794ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXX.X...................................................] duration: 678.661508ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:48:45 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 678.687937ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 378.835695ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:47:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.879557ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XXXXXXXX.X..................................................] duration: 604.85604ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:47:40 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 604.872922ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 427.685943ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:47:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.704508ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXX.X.................................................] duration: 598.205954ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:47:21 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 598.224168ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 456.460473ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:47:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.499786ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 494.650844ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:47:02 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 494.667565ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXX.X................................................] duration: 589.618128ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:46:45 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 589.64078ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 491.873585ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.898713ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 548.921232ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:46:18 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 548.937663ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X...............................................] duration: 843.308305ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:46:11 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 843.346777ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 423.262998ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.282766ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X..............................................] duration: 592.456802ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:45:26 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 592.475558ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 475.692807ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:45:22 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 475.712214ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 505.312069ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.331164ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 730.004919ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:44:30 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 730.019056ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)