Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X....XXX] duration: 30.000673682s FAIL (2025-08-07 23:16:50 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 778582181 (6h47m34.365638402s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778802352 (6h47m34.365858573s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779022393 (6h47m34.366078604s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779118041 (6h47m34.366174282s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779253655 (6h47m34.366309876s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779477572 (6h47m34.366533784s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779672236 (6h47m34.366728477s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779697914 (6h47m34.366754135s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780260564 (6h47m34.367316805s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780779803 (6h47m34.367836044s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.781487684 (6h47m34.368543936s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.858608298 (6h47m34.445664509s) [upload=6969]: Next returned nil segment Aug 7 23:14:20.858630338 (6h47m34.445686559s) [upload=6969]: Waiting for error group managing segments... Aug 7 23:14:20.867703544 (6h47m34.454759765s) [noise=true , address=45.91.116.6:4973 , piece_id=JNT2NU7DK7LVLBF4CK3Q3H2L5VNAEYZTFGEAAEYTRSN5QNLCMA2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869347483 (6h47m34.456403704s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869615523 (6h47m34.456671734s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.870768575 (6h47m34.457824796s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.872453060 (6h47m34.459509281s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.873530410 (6h47m34.460586621s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.874615627 (6h47m34.461671838s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.875179007 (6h47m34.462235218s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.880399252 (6h47m34.467455473s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.883816901 (6h47m34.470873122s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.886328790 (6h47m34.473385001s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.889471356 (6h47m34.476527577s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893860889 (6h47m34.48091711s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893882650 (6h47m34.48093886s) [seg_pos={0 0}, upload=6969]: Segment reached optimal threshold of 13 pieces. Aug 7 23:14:20.894050533 (6h47m34.481106754s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Aug 7 23:14:20.894105426 (6h47m34.481161656s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection Aug 7 23:14:20.894188311 (6h47m34.481244532s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:36452->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:14:20.894198529 (6h47m34.48125475s) [seg_pos={0 0}, upload=6969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:14:20.894279671 (6h47m34.481335882s) [upload=6969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:14:20.894283668 (6h47m34.481339889s) [upload=6969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:20.936620403 (6h47m34.523676654s) [upload=6969]: Done uploading segments... Aug 7 23:14:20.936636754 (6h47m34.523692975s) [upload=6969]: Upload finished. err: <nil> Aug 7 23:14:44.515675412 (6h47m58.102731643s) [upload=6976]: Starting upload Aug 7 23:14:44.515752076 (6h47m58.102808317s) [upload=6976]: Done starting upload Aug 7 23:14:44.515823649 (6h47m58.10287991s) [upload=6976]: Uploading segments... Aug 7 23:14:44.516164195 (6h47m58.103220436s) [upload=6976]: Got next segment. Inline: false Aug 7 23:14:44.516248702 (6h47m58.103304944s) [upload=6976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:44.516256998 (6h47m58.103313239s) [upload=6976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:14.501722064 (6h48m28.088778285s) [upload=6976]: Next segment err: context deadline exceeded Aug 7 23:15:14.501867856 (6h48m28.088924078s) [upload=6976]: Done uploading segments... Aug 7 23:15:14.501885520 (6h48m28.088941731s) [upload=6976]: Upload finished. err: context deadline exceeded Aug 7 23:15:50.351511032 (6h49m3.938567273s) [upload=6996]: Starting upload Aug 7 23:15:50.351596221 (6h49m3.938652472s) [upload=6996]: Done starting upload Aug 7 23:15:50.351682441 (6h49m3.938738683s) [upload=6996]: Uploading segments... Aug 7 23:15:50.351994003 (6h49m3.939050244s) [upload=6996]: Got next segment. Inline: false Aug 7 23:15:50.352036763 (6h49m3.939093004s) [upload=6996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:50.352042523 (6h49m3.939098765s) [upload=6996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:20.337163793 (6h49m33.924220044s) [upload=6996]: Next segment err: context deadline exceeded Aug 7 23:16:20.337425733 (6h49m33.924481983s) [upload=6996]: Done uploading segments... Aug 7 23:16:20.337475926 (6h49m33.924532177s) [upload=6996]: Upload finished. err: context deadline exceeded Aug 7 23:16:20.420371668 (6h49m34.007427909s) [upload=7003]: Starting upload Aug 7 23:16:20.420456566 (6h49m34.007512807s) [upload=7003]: Done starting upload Aug 7 23:16:20.420524824 (6h49m34.007581066s) [upload=7003]: Uploading segments... Aug 7 23:16:20.420851844 (6h49m34.007908086s) [upload=7003]: Got next segment. Inline: false Aug 7 23:16:20.420905824 (6h49m34.007962066s) [upload=7003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.420912007 (6h49m34.007968247s) [upload=7003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:50.407120057 (6h50m3.994176297s) [upload=7003]: Next segment err: context deadline exceeded Aug 7 23:16:50.407194757 (6h50m3.994250997s) [upload=7003]: Done uploading segments... Aug 7 23:16:50.407241563 (6h50m3.994297814s) [upload=7003]: Upload finished. err: context deadline exceeded Duration: 30.000693157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.001409721s FAIL (2025-08-07 23:16:48 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001424098s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 909.160305ms FAIL (2025-08-07 23:16:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 909.191263ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 185.692702ms FAIL (2025-08-07 23:16:35 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 185.713171ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 16.460655ms FAIL (2025-08-07 23:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 16.480473ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 201.85503ms FAIL (2025-08-07 23:16:26 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Duration: 201.877993ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X....XX] duration: 30.000859878s FAIL (2025-08-07 23:16:20 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== {0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.777883617 (6h47m34.364939858s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778085284 (6h47m34.365141496s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778341011 (6h47m34.365397222s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778504276 (6h47m34.365560517s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778582181 (6h47m34.365638402s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778802352 (6h47m34.365858573s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779022393 (6h47m34.366078604s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779118041 (6h47m34.366174282s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779253655 (6h47m34.366309876s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779477572 (6h47m34.366533784s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779672236 (6h47m34.366728477s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779697914 (6h47m34.366754135s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780260564 (6h47m34.367316805s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780779803 (6h47m34.367836044s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.781487684 (6h47m34.368543936s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.858608298 (6h47m34.445664509s) [upload=6969]: Next returned nil segment Aug 7 23:14:20.858630338 (6h47m34.445686559s) [upload=6969]: Waiting for error group managing segments... Aug 7 23:14:20.867703544 (6h47m34.454759765s) [noise=true , address=45.91.116.6:4973 , piece_id=JNT2NU7DK7LVLBF4CK3Q3H2L5VNAEYZTFGEAAEYTRSN5QNLCMA2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869347483 (6h47m34.456403704s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869615523 (6h47m34.456671734s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.870768575 (6h47m34.457824796s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.872453060 (6h47m34.459509281s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.873530410 (6h47m34.460586621s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.874615627 (6h47m34.461671838s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.875179007 (6h47m34.462235218s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.880399252 (6h47m34.467455473s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.883816901 (6h47m34.470873122s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.886328790 (6h47m34.473385001s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.889471356 (6h47m34.476527577s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893860889 (6h47m34.48091711s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893882650 (6h47m34.48093886s) [seg_pos={0 0}, upload=6969]: Segment reached optimal threshold of 13 pieces. Aug 7 23:14:20.894050533 (6h47m34.481106754s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Aug 7 23:14:20.894105426 (6h47m34.481161656s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection Aug 7 23:14:20.894188311 (6h47m34.481244532s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:36452->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:14:20.894198529 (6h47m34.48125475s) [seg_pos={0 0}, upload=6969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:14:20.894279671 (6h47m34.481335882s) [upload=6969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:14:20.894283668 (6h47m34.481339889s) [upload=6969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:20.936620403 (6h47m34.523676654s) [upload=6969]: Done uploading segments... Aug 7 23:14:20.936636754 (6h47m34.523692975s) [upload=6969]: Upload finished. err: <nil> Aug 7 23:14:44.515675412 (6h47m58.102731643s) [upload=6976]: Starting upload Aug 7 23:14:44.515752076 (6h47m58.102808317s) [upload=6976]: Done starting upload Aug 7 23:14:44.515823649 (6h47m58.10287991s) [upload=6976]: Uploading segments... Aug 7 23:14:44.516164195 (6h47m58.103220436s) [upload=6976]: Got next segment. Inline: false Aug 7 23:14:44.516248702 (6h47m58.103304944s) [upload=6976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:44.516256998 (6h47m58.103313239s) [upload=6976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:14.501722064 (6h48m28.088778285s) [upload=6976]: Next segment err: context deadline exceeded Aug 7 23:15:14.501867856 (6h48m28.088924078s) [upload=6976]: Done uploading segments... Aug 7 23:15:14.501885520 (6h48m28.088941731s) [upload=6976]: Upload finished. err: context deadline exceeded Aug 7 23:15:50.351511032 (6h49m3.938567273s) [upload=6996]: Starting upload Aug 7 23:15:50.351596221 (6h49m3.938652472s) [upload=6996]: Done starting upload Aug 7 23:15:50.351682441 (6h49m3.938738683s) [upload=6996]: Uploading segments... Aug 7 23:15:50.351994003 (6h49m3.939050244s) [upload=6996]: Got next segment. Inline: false Aug 7 23:15:50.352036763 (6h49m3.939093004s) [upload=6996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:50.352042523 (6h49m3.939098765s) [upload=6996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:20.337163793 (6h49m33.924220044s) [upload=6996]: Next segment err: context deadline exceeded Aug 7 23:16:20.337425733 (6h49m33.924481983s) [upload=6996]: Done uploading segments... Aug 7 23:16:20.337475926 (6h49m33.924532177s) [upload=6996]: Upload finished. err: context deadline exceeded Duration: 30.000909931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.001316166s FAIL (2025-08-07 23:16:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em_ObjectCommit Aug 7 23:07:43.649755305 (6h40m57.236811607s) [upload=6854]: Done uploading segments... Aug 7 23:07:43.649779701 (6h40m57.236835953s) [upload=6854]: Upload finished. err: <nil> Aug 7 23:08:56.878235045 (6h42m10.465291266s) [upload=6873]: Starting upload Aug 7 23:08:56.878316127 (6h42m10.465372348s) [upload=6873]: Done starting upload Aug 7 23:08:56.878378183 (6h42m10.465434404s) [upload=6873]: Uploading segments... Aug 7 23:08:56.878424890 (6h42m10.46548114s) [upload=6873]: Got next segment. Inline: true Aug 7 23:08:56.878430340 (6h42m10.465486561s) [upload=6873]: Waiting for error group managing segments... Aug 7 23:08:56.878474843 (6h42m10.465531064s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:56.878479702 (6h42m10.465535923s) [upload=6873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:56.878483188 (6h42m10.465539409s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:56.941171742 (6h42m10.528227963s) [upload=6873]: Done uploading segments... Aug 7 23:08:56.941188142 (6h42m10.528244354s) [upload=6873]: Upload finished. err: <nil> Aug 7 23:09:42.944491406 (6h42m56.531547628s) [upload=6888]: Starting upload Aug 7 23:09:42.944574772 (6h42m56.531630993s) [upload=6888]: Done starting upload Aug 7 23:09:42.944630365 (6h42m56.531686576s) [upload=6888]: Uploading segments... Aug 7 23:09:42.944668176 (6h42m56.531724397s) [upload=6888]: Got next segment. Inline: true Aug 7 23:09:42.944673766 (6h42m56.531729977s) [upload=6888]: Waiting for error group managing segments... Aug 7 23:09:42.944715465 (6h42m56.531771686s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:42.944720784 (6h42m56.531777005s) [upload=6888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:42.944724171 (6h42m56.531780392s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:43.008585323 (6h42m56.595641534s) [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001329481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000799059s FAIL (2025-08-07 23:16:10 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000818977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X......................................................XXXXX] duration: 30.000941106s FAIL (2025-08-07 23:16:09 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000965511s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X.XX.] duration: 484.78228ms FAIL (2025-08-07 23:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.799764ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 280.347903ms FAIL (2025-08-07 23:16:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 280.379663ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000607827s FAIL (2025-08-07 23:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 486167538 (6h47m1.073223769s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.486657492 (6h47m1.073713733s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.486694452 (6h47m1.073750683s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487035368 (6h47m1.074091609s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487147918 (6h47m1.074204149s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487434914 (6h47m1.074491205s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487589051 (6h47m1.074645282s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487895814 (6h47m1.074952045s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488059319 (6h47m1.07511555s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488340534 (6h47m1.075396765s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488536069 (6h47m1.0755923s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488721475 (6h47m1.075777716s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489079183 (6h47m1.076135424s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489506110 (6h47m1.07656235s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489945269 (6h47m1.077001511s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490429594 (6h47m1.077485834s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490819892 (6h47m1.077876133s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.574630371 (6h47m1.161686592s) [upload=6952]: Next returned nil segment Aug 7 23:13:47.574641062 (6h47m1.161697283s) [upload=6952]: Waiting for error group managing segments... Aug 7 23:13:47.587541340 (6h47m1.174597562s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.592242405 (6h47m1.179298626s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.599971292 (6h47m1.187027503s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.607959633 (6h47m1.195015854s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.611555756 (6h47m1.198611967s) [noise=true , address=104.36.82.58:4279 , piece_id=AH5MJW4W346CNJN6LPTKW3FNQV6SUWP5JC47HDH5SRU4D7OWI43Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.626811131 (6h47m1.213867352s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.637742776 (6h47m1.224799007s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639233228 (6h47m1.226289449s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639680683 (6h47m1.226736904s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.646508387 (6h47m1.233564599s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.669751322 (6h47m1.256807543s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.722599178 (6h47m1.309655429s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.724255018 (6h47m1.31131124s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740407329 (6h47m1.32746358s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740458694 (6h47m1.327514935s) [seg_pos={0 0}, upload=6952]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:47.740645193 (6h47m1.327701414s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:47.740952506 (6h47m1.328008727s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: read tcp 5.161.89.252:25140->64.49.13.169:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741048936 (6h47m1.328105147s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122k7b6wa2JhSXMVAhF4qPKA48cWkh3DZRyWuwPEt7Vp3ciofPF): protocol: read tcp 5.161.89.252:36630->104.36.82.61:4407: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741097086 (6h47m1.328153327s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:22104->193.149.164.143:4697: use of closed network connection; conn closed: op Aug 7 23:13:47.741147290 (6h47m1.32820353s) [seg_pos={0 0}, upload=6952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:47.741286780 (6h47m1.328343021s) [upload=6952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:47.741297580 (6h47m1.328353832s) [upload=6952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:59.351303634 (6h48m12.938359875s) [upload=6952]: Done uploading segments... Aug 7 23:14:59.351331766 (6h48m12.938388007s) [upload=6952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:59.433768531 (6h48m13.020824773s) [upload=6982]: Starting upload Aug 7 23:14:59.433862216 (6h48m13.020918457s) [upload=6982]: Done starting upload Aug 7 23:14:59.434026252 (6h48m13.021082504s) [upload=6982]: Uploading segments... Aug 7 23:14:59.434316203 (6h48m13.021372445s) [upload=6982]: Got next segment. Inline: false Aug 7 23:14:59.434383048 (6h48m13.02143929s) [upload=6982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.434389380 (6h48m13.021445622s) [upload=6982]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:59.416073113 (6h49m13.003129334s) [upload=6982]: Next segment err: context deadline exceeded Aug 7 23:15:59.416143385 (6h49m13.003199606s) [upload=6982]: Done uploading segments... Aug 7 23:15:59.416158513 (6h49m13.003214734s) [upload=6982]: Upload finished. err: context deadline exceeded Duration: 1m0.00062569s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 12.738276ms target: https://us1.storj.io/login FAIL (2025-08-07 23:15:56 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 12.761781ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.002302571s FAIL (2025-08-07 23:15:53 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.002324834s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.......................................................XXX] duration: 1m0.001528617s FAIL (2025-08-07 23:15:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001543766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 892.920441ms FAIL (2025-08-07 23:15:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 892.94661ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 30.000989899s FAIL (2025-08-07 23:15:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001011261s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000339848s FAIL (2025-08-07 23:15:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000356689s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000588613s FAIL (2025-08-07 23:15:20 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000616194s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X......................................................XXXX] duration: 30.001369635s FAIL (2025-08-07 23:15:15 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001391817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.000591462s FAIL (2025-08-07 23:15:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:48.578044013 (6h40m2.165100234s) [upload=6833]: Done uploading segments... Aug 7 23:06:48.578050885 (6h40m2.165107106s) [upload=6833]: Upload finished. err: <nil> Aug 7 23:07:43.592700568 (6h40m57.179756809s) [upload=6854]: Starting upload Aug 7 23:07:43.592773794 (6h40m57.179830045s) [upload=6854]: Done starting upload Aug 7 23:07:43.592853843 (6h40m57.179910084s) [upload=6854]: Uploading segments... Aug 7 23:07:43.592922552 (6h40m57.179978813s) [upload=6854]: Got next segment. Inline: true Aug 7 23:07:43.592928533 (6h40m57.179984773s) [upload=6854]: Waiting for error group managing segments... Aug 7 23:07:43.592972054 (6h40m57.180028305s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:43.592977975 (6h40m57.180034216s) [upload=6854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:43.593034962 (6h40m57.180091213s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:43.649755305 (6h40m57.236811607s) [upload=6854]: Done uploading segments... Aug 7 23:07:43.649779701 (6h40m57.236835953s) [upload=6854]: Upload finished. err: <nil> Aug 7 23:08:56.878235045 (6h42m10.465291266s) [upload=6873]: Starting upload Aug 7 23:08:56.878316127 (6h42m10.465372348s) [upload=6873]: Done starting upload Aug 7 23:08:56.878378183 (6h42m10.465434404s) [upload=6873]: Uploading segments... Aug 7 23:08:56.878424890 (6h42m10.46548114s) [upload=6873]: Got next segment. Inline: true Aug 7 23:08:56.878430340 (6h42m10.465486561s) [upload=6873]: Waiting for error group managing segments... Aug 7 23:08:56.878474843 (6h42m10.465531064s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:56.878479702 (6h42m10.465535923s) [upload=6873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:56.878483188 (6h42m10.465539409s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:56.941171742 (6h42m10.528227963s) [upload=6873]: Done uploading segments... Aug 7 23:08:56.941188142 (6h42m10.528244354s) [upload=6873]: Upload finished. err: <nil> Aug 7 23:09:42.944491406 (6h42m56.531547628s) [upload=6888]: Starting upload Aug 7 23:09:42.944574772 (6h42m56.531630993s) [upload=6888]: Done starting upload Aug 7 23:09:42.944630365 (6h42m56.531686576s) [upload=6888]: Uploading segments... Aug 7 23:09:42.944668176 (6h42m56.531724397s) [upload=6888]: Got next segment. Inline: true Aug 7 23:09:42.944673766 (6h42m56.531729977s) [upload=6888]: Waiting for error group managing segments... Aug 7 23:09:42.944715465 (6h42m56.531771686s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:42.944720784 (6h42m56.531777005s) [upload=6888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:42.944724171 (6h42m56.531780392s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:43.008585323 (6h42m56.595641534s) [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.00060657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.001396704s FAIL (2025-08-07 23:15:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g upload Aug 7 23:14:20.752798114 (6h47m34.339854354s) [upload=6969]: Done starting upload Aug 7 23:14:20.752861752 (6h47m34.339917963s) [upload=6969]: Uploading segments... Aug 7 23:14:20.753169025 (6h47m34.340225266s) [upload=6969]: Got next segment. Inline: false Aug 7 23:14:20.753241391 (6h47m34.340297632s) [upload=6969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:20.753246791 (6h47m34.340303032s) [upload=6969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:20.777732626 (6h47m34.364788847s) [seg_pos={0 0}, upload=6969]: Begin upload segment... Aug 7 23:14:20.777765948 (6h47m34.364822169s) [seg_pos={0 0}, upload=6969]: Done begin upload segment. Aug 7 23:14:20.777785724 (6h47m34.364841936s) [noise=true , address=45.91.116.6:4973 , piece_id=JNT2NU7DK7LVLBF4CK3Q3H2L5VNAEYZTFGEAAEYTRSN5QNLCMA2Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.777883617 (6h47m34.364939858s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778085284 (6h47m34.365141496s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778341011 (6h47m34.365397222s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778504276 (6h47m34.365560517s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778582181 (6h47m34.365638402s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778802352 (6h47m34.365858573s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779022393 (6h47m34.366078604s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779118041 (6h47m34.366174282s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779253655 (6h47m34.366309876s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779477572 (6h47m34.366533784s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779672236 (6h47m34.366728477s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779697914 (6h47m34.366754135s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780260564 (6h47m34.367316805s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780779803 (6h47m34.367836044s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.781487684 (6h47m34.368543936s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.858608298 (6h47m34.445664509s) [upload=6969]: Next returned nil segment Aug 7 23:14:20.858630338 (6h47m34.445686559s) [upload=6969]: Waiting for error group managing segments... Aug 7 23:14:20.867703544 (6h47m34.454759765s) [noise=true , address=45.91.116.6:4973 , piece_id=JNT2NU7DK7LVLBF4CK3Q3H2L5VNAEYZTFGEAAEYTRSN5QNLCMA2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869347483 (6h47m34.456403704s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869615523 (6h47m34.456671734s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.870768575 (6h47m34.457824796s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.872453060 (6h47m34.459509281s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.873530410 (6h47m34.460586621s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.874615627 (6h47m34.461671838s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.875179007 (6h47m34.462235218s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.880399252 (6h47m34.467455473s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.883816901 (6h47m34.470873122s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.886328790 (6h47m34.473385001s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.889471356 (6h47m34.476527577s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893860889 (6h47m34.48091711s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893882650 (6h47m34.48093886s) [seg_pos={0 0}, upload=6969]: Segment reached optimal threshold of 13 pieces. Aug 7 23:14:20.894050533 (6h47m34.481106754s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Aug 7 23:14:20.894105426 (6h47m34.481161656s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection Aug 7 23:14:20.894188311 (6h47m34.481244532s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:36452->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:14:20.894198529 (6h47m34.48125475s) [seg_pos={0 0}, upload=6969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:14:20.894279671 (6h47m34.481335882s) [upload=6969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:14:20.894283668 (6h47m34.481339889s) [upload=6969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:20.936620403 (6h47m34.523676654s) [upload=6969]: Done uploading segments... Aug 7 23:14:20.936636754 (6h47m34.523692975s) [upload=6969]: Upload finished. err: <nil> Aug 7 23:14:44.515675412 (6h47m58.102731643s) [upload=6976]: Starting upload Aug 7 23:14:44.515752076 (6h47m58.102808317s) [upload=6976]: Done starting upload Aug 7 23:14:44.515823649 (6h47m58.10287991s) [upload=6976]: Uploading segments... Aug 7 23:14:44.516164195 (6h47m58.103220436s) [upload=6976]: Got next segment. Inline: false Aug 7 23:14:44.516248702 (6h47m58.103304944s) [upload=6976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:44.516256998 (6h47m58.103313239s) [upload=6976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:14.501722064 (6h48m28.088778285s) [upload=6976]: Next segment err: context deadline exceeded Aug 7 23:15:14.501867856 (6h48m28.088924078s) [upload=6976]: Done uploading segments... Aug 7 23:15:14.501885520 (6h48m28.088941731s) [upload=6976]: Upload finished. err: context deadline exceeded Duration: 30.001420519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.000814046s FAIL (2025-08-07 23:15:07 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== stItem_ObjectCommit Aug 7 23:09:07.161521184 (6h42m20.748577405s) [upload=6875]: Done uploading segments... Aug 7 23:09:07.161537344 (6h42m20.748593565s) [upload=6875]: Upload finished. err: <nil> Aug 7 23:09:32.587106585 (6h42m46.174162806s) [upload=6885]: Starting upload Aug 7 23:09:32.587221680 (6h42m46.174277901s) [upload=6885]: Done starting upload Aug 7 23:09:32.587315766 (6h42m46.174371977s) [upload=6885]: Uploading segments... Aug 7 23:09:32.587429769 (6h42m46.174486019s) [upload=6885]: Got next segment. Inline: true Aug 7 23:09:32.587436501 (6h42m46.174492722s) [upload=6885]: Waiting for error group managing segments... Aug 7 23:09:32.587485553 (6h42m46.174541764s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:32.587491184 (6h42m46.174547405s) [upload=6885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:32.587495031 (6h42m46.174551252s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00083768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m41.476896777s FAIL (2025-08-07 23:14:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 251408s) [upload=6952]: Done starting upload Aug 7 23:13:17.893277030 (6h46m31.480333241s) [upload=6952]: Uploading segments... Aug 7 23:13:17.893427069 (6h46m31.48048328s) [upload=6952]: Got next segment. Inline: false Aug 7 23:13:17.893474167 (6h46m31.480530379s) [upload=6952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:17.893479297 (6h46m31.480535508s) [upload=6952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:47.486043867 (6h47m1.073100099s) [seg_pos={0 0}, upload=6952]: Begin upload segment... Aug 7 23:13:47.486102436 (6h47m1.073158677s) [seg_pos={0 0}, upload=6952]: Done begin upload segment. Aug 7 23:13:47.486136670 (6h47m1.073192912s) [noise=true , address=104.36.82.58:4279 , piece_id=AH5MJW4W346CNJN6LPTKW3FNQV6SUWP5JC47HDH5SRU4D7OWI43Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.486167538 (6h47m1.073223769s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.486657492 (6h47m1.073713733s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.486694452 (6h47m1.073750683s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487035368 (6h47m1.074091609s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487147918 (6h47m1.074204149s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487434914 (6h47m1.074491205s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487589051 (6h47m1.074645282s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487895814 (6h47m1.074952045s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488059319 (6h47m1.07511555s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488340534 (6h47m1.075396765s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488536069 (6h47m1.0755923s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488721475 (6h47m1.075777716s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489079183 (6h47m1.076135424s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489506110 (6h47m1.07656235s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489945269 (6h47m1.077001511s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490429594 (6h47m1.077485834s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490819892 (6h47m1.077876133s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.574630371 (6h47m1.161686592s) [upload=6952]: Next returned nil segment Aug 7 23:13:47.574641062 (6h47m1.161697283s) [upload=6952]: Waiting for error group managing segments... Aug 7 23:13:47.587541340 (6h47m1.174597562s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.592242405 (6h47m1.179298626s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.599971292 (6h47m1.187027503s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.607959633 (6h47m1.195015854s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.611555756 (6h47m1.198611967s) [noise=true , address=104.36.82.58:4279 , piece_id=AH5MJW4W346CNJN6LPTKW3FNQV6SUWP5JC47HDH5SRU4D7OWI43Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.626811131 (6h47m1.213867352s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.637742776 (6h47m1.224799007s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639233228 (6h47m1.226289449s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639680683 (6h47m1.226736904s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.646508387 (6h47m1.233564599s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.669751322 (6h47m1.256807543s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.722599178 (6h47m1.309655429s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.724255018 (6h47m1.31131124s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740407329 (6h47m1.32746358s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740458694 (6h47m1.327514935s) [seg_pos={0 0}, upload=6952]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:47.740645193 (6h47m1.327701414s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:47.740952506 (6h47m1.328008727s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: read tcp 5.161.89.252:25140->64.49.13.169:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741048936 (6h47m1.328105147s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122k7b6wa2JhSXMVAhF4qPKA48cWkh3DZRyWuwPEt7Vp3ciofPF): protocol: read tcp 5.161.89.252:36630->104.36.82.61:4407: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741097086 (6h47m1.328153327s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:22104->193.149.164.143:4697: use of closed network connection; conn closed: op Aug 7 23:13:47.741147290 (6h47m1.32820353s) [seg_pos={0 0}, upload=6952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:47.741286780 (6h47m1.328343021s) [upload=6952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:47.741297580 (6h47m1.328353832s) [upload=6952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:59.351303634 (6h48m12.938359875s) [upload=6952]: Done uploading segments... Aug 7 23:14:59.351331766 (6h48m12.938388007s) [upload=6952]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m41.476929819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001177575s FAIL (2025-08-07 23:14:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001195178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000981455s FAIL (2025-08-07 23:14:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00100054s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................................XX] duration: 1m0.000551352s FAIL (2025-08-07 23:14:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00056615s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 23.918587ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:14:43 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 23.939335ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 14.219713ms target: https://us1.storj.io/login FAIL (2025-08-07 23:14:41 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 14.262873ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000847371s FAIL (2025-08-07 23:14:34 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:26.662838039 (6h41m40.249894279s) [upload=6863]: Done uploading segments... Aug 7 23:08:26.662853056 (6h41m40.249909298s) [upload=6863]: Upload finished. err: <nil> Aug 7 23:09:07.080522893 (6h42m20.667579114s) [upload=6875]: Starting upload Aug 7 23:09:07.080579920 (6h42m20.667636141s) [upload=6875]: Done starting upload Aug 7 23:09:07.080644431 (6h42m20.667700642s) [upload=6875]: Uploading segments... Aug 7 23:09:07.080682552 (6h42m20.667738773s) [upload=6875]: Got next segment. Inline: true Aug 7 23:09:07.080688112 (6h42m20.667744334s) [upload=6875]: Waiting for error group managing segments... Aug 7 23:09:07.080725373 (6h42m20.667781594s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:07.080729751 (6h42m20.667785972s) [upload=6875]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:07.080732115 (6h42m20.667788326s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:07.161521184 (6h42m20.748577405s) [upload=6875]: Done uploading segments... Aug 7 23:09:07.161537344 (6h42m20.748593565s) [upload=6875]: Upload finished. err: <nil> Aug 7 23:09:32.587106585 (6h42m46.174162806s) [upload=6885]: Starting upload Aug 7 23:09:32.587221680 (6h42m46.174277901s) [upload=6885]: Done starting upload Aug 7 23:09:32.587315766 (6h42m46.174371977s) [upload=6885]: Uploading segments... Aug 7 23:09:32.587429769 (6h42m46.174486019s) [upload=6885]: Got next segment. Inline: true Aug 7 23:09:32.587436501 (6h42m46.174492722s) [upload=6885]: Waiting for error group managing segments... Aug 7 23:09:32.587485553 (6h42m46.174541764s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:32.587491184 (6h42m46.174547405s) [upload=6885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:32.587495031 (6h42m46.174551252s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.000867729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.140007021s FAIL (2025-08-07 23:14:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 1.140035665s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.........................................................X.] duration: 21.580831221s FAIL (2025-08-07 23:14:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.580855246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 376.643127ms FAIL (2025-08-07 23:14:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== h43m23.967460908s) [upload=6897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:10.380410569 (6h43m23.96746681s) [upload=6897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:10.423498627 (6h43m24.010554838s) [upload=6897]: Done uploading segments... Aug 7 23:10:10.423508945 (6h43m24.010565157s) [upload=6897]: Upload finished. err: <nil> Aug 7 23:10:28.548503936 (6h43m42.135560176s) [upload=6903]: Starting upload Aug 7 23:10:28.548587942 (6h43m42.135644183s) [upload=6903]: Done starting upload Aug 7 23:10:28.548687107 (6h43m42.135743348s) [upload=6903]: Uploading segments... Aug 7 23:10:28.548952603 (6h43m42.136008854s) [upload=6903]: Got next segment. Inline: false Aug 7 23:10:28.548991997 (6h43m42.136048238s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.548997387 (6h43m42.136053628s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:10:28.575505557 (6h43m42.162561808s) [seg_pos={0 0}, upload=6903]: Begin upload segment... Aug 7 23:10:28.575543988 (6h43m42.162600229s) [seg_pos={0 0}, upload=6903]: Done begin upload segment. Aug 7 23:10:28.575570117 (6h43m42.162626358s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.575623586 (6h43m42.162679827s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576079848 (6h43m42.163136089s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576336698 (6h43m42.163392938s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576464947 (6h43m42.163521189s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576739170 (6h43m42.16379541s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576930076 (6h43m42.163986316s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577101956 (6h43m42.164158207s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577333558 (6h43m42.164389799s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577523042 (6h43m42.164579293s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577694552 (6h43m42.164750793s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577950891 (6h43m42.165007132s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578056468 (6h43m42.165112719s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578412783 (6h43m42.165469065s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578497261 (6h43m42.165553512s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578896225 (6h43m42.165952466s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.726482559 (6h43m42.3135388s) [upload=6903]: Next returned nil segment Aug 7 23:10:28.726499210 (6h43m42.313555452s) [upload=6903]: Waiting for error group managing segments... Aug 7 23:10:28.733585057 (6h43m42.320641298s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.734493031 (6h43m42.321549272s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735107198 (6h43m42.322163439s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735334542 (6h43m42.322390784s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735677101 (6h43m42.322733343s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736304363 (6h43m42.323360613s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736733653 (6h43m42.323789894s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.737010621 (6h43m42.324066872s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.738308202 (6h43m42.325364454s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.739491822 (6h43m42.326548064s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.740628304 (6h43m42.327684545s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.741118448 (6h43m42.328174699s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743301704 (6h43m42.330357955s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743318215 (6h43m42.330374466s) [seg_pos={0 0}, upload=6903]: Segment reached optimal threshold of 13 pieces. Aug 7 23:10:28.743478223 (6h43m42.330534464s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743579422 (6h43m42.330635673s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Aug 7 23:10:28.743685970 (6h43m42.330742212s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:64866->104.36.82.15:4257: use of closed network connection; context canceled Aug 7 23:10:28.743721447 (6h43m42.330777688s) [seg_pos={0 0}, upload=6903]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Aug 7 23:10:28.743791097 (6h43m42.330847338s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:28.743796517 (6h43m42.330852758s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.784620930 (6h43m42.371677172s) [upload=6903]: Done uploading segments... Aug 7 23:10:28.784637271 (6h43m42.371693512s) [upload=6903]: Upload finished. err: <nil> Aug 7 23:11:14.178732655 (6h44m27.765788876s) [upload=6917]: Starting upload Aug 7 23:11:14.178789762 (6h44m27.765845983s) [upload=6917]: Done starting upload Aug 7 23:11:14.178845816 (6h44m27.765902038s) [upload=6917]: Uploading segments... Aug 7 23:11:14.179029189 (6h44m27.76608541s) [upload=6917]: Got next segment. Inline: false Aug 7 23:11:14.179063893 (6h44m27.766120104s) [upload=6917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:14.179069093 (6h44m27.766125314s) [upload=6917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:44.165125191 (6h44m57.752181432s) [upload=6917]: Next segment err: context deadline exceeded Aug 7 23:11:44.165321557 (6h44m57.752377798s) [upload=6917]: Done uploading segments... Aug 7 23:11:44.165369758 (6h44m57.752425998s) [upload=6917]: Upload finished. err: context deadline exceeded Duration: 376.662574ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X......................................................XXX] duration: 30.00104962s FAIL (2025-08-07 23:14:15 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001074296s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.001488582s FAIL (2025-08-07 23:14:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 65 (6h39m6.588389176s) [upload=6817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:53.059652995 (6h39m6.646709206s) [upload=6817]: Done uploading segments... Aug 7 23:05:53.059668294 (6h39m6.646724515s) [upload=6817]: Upload finished. err: <nil> Aug 7 23:06:48.439632034 (6h40m2.026688255s) [upload=6833]: Starting upload Aug 7 23:06:48.439692607 (6h40m2.026748818s) [upload=6833]: Done starting upload Aug 7 23:06:48.439737180 (6h40m2.026793391s) [upload=6833]: Uploading segments... Aug 7 23:06:48.439763910 (6h40m2.026820122s) [upload=6833]: Got next segment. Inline: true Aug 7 23:06:48.439767437 (6h40m2.026823658s) [upload=6833]: Waiting for error group managing segments... Aug 7 23:06:48.439799687 (6h40m2.026855908s) [upload=6833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:48.439803343 (6h40m2.026859564s) [upload=6833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:48.439805758 (6h40m2.026861979s) [upload=6833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:48.578044013 (6h40m2.165100234s) [upload=6833]: Done uploading segments... Aug 7 23:06:48.578050885 (6h40m2.165107106s) [upload=6833]: Upload finished. err: <nil> Aug 7 23:07:43.592700568 (6h40m57.179756809s) [upload=6854]: Starting upload Aug 7 23:07:43.592773794 (6h40m57.179830045s) [upload=6854]: Done starting upload Aug 7 23:07:43.592853843 (6h40m57.179910084s) [upload=6854]: Uploading segments... Aug 7 23:07:43.592922552 (6h40m57.179978813s) [upload=6854]: Got next segment. Inline: true Aug 7 23:07:43.592928533 (6h40m57.179984773s) [upload=6854]: Waiting for error group managing segments... Aug 7 23:07:43.592972054 (6h40m57.180028305s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:43.592977975 (6h40m57.180034216s) [upload=6854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:43.593034962 (6h40m57.180091213s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:43.649755305 (6h40m57.236811607s) [upload=6854]: Done uploading segments... Aug 7 23:07:43.649779701 (6h40m57.236835953s) [upload=6854]: Upload finished. err: <nil> Aug 7 23:08:56.878235045 (6h42m10.465291266s) [upload=6873]: Starting upload Aug 7 23:08:56.878316127 (6h42m10.465372348s) [upload=6873]: Done starting upload Aug 7 23:08:56.878378183 (6h42m10.465434404s) [upload=6873]: Uploading segments... Aug 7 23:08:56.878424890 (6h42m10.46548114s) [upload=6873]: Got next segment. Inline: true Aug 7 23:08:56.878430340 (6h42m10.465486561s) [upload=6873]: Waiting for error group managing segments... Aug 7 23:08:56.878474843 (6h42m10.465531064s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:56.878479702 (6h42m10.465535923s) [upload=6873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:56.878483188 (6h42m10.465539409s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:56.941171742 (6h42m10.528227963s) [upload=6873]: Done uploading segments... Aug 7 23:08:56.941188142 (6h42m10.528244354s) [upload=6873]: Upload finished. err: <nil> Aug 7 23:09:42.944491406 (6h42m56.531547628s) [upload=6888]: Starting upload Aug 7 23:09:42.944574772 (6h42m56.531630993s) [upload=6888]: Done starting upload Aug 7 23:09:42.944630365 (6h42m56.531686576s) [upload=6888]: Uploading segments... Aug 7 23:09:42.944668176 (6h42m56.531724397s) [upload=6888]: Got next segment. Inline: true Aug 7 23:09:42.944673766 (6h42m56.531729977s) [upload=6888]: Waiting for error group managing segments... Aug 7 23:09:42.944715465 (6h42m56.531771686s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:42.944720784 (6h42m56.531777005s) [upload=6888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:42.944724171 (6h42m56.531780392s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:43.008585323 (6h42m56.595641534s) [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001542874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 389.877004ms FAIL (2025-08-07 23:14:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.904074ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 778.131216ms FAIL (2025-08-07 23:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== h43m23.967460908s) [upload=6897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:10.380410569 (6h43m23.96746681s) [upload=6897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:10.423498627 (6h43m24.010554838s) [upload=6897]: Done uploading segments... Aug 7 23:10:10.423508945 (6h43m24.010565157s) [upload=6897]: Upload finished. err: <nil> Aug 7 23:10:28.548503936 (6h43m42.135560176s) [upload=6903]: Starting upload Aug 7 23:10:28.548587942 (6h43m42.135644183s) [upload=6903]: Done starting upload Aug 7 23:10:28.548687107 (6h43m42.135743348s) [upload=6903]: Uploading segments... Aug 7 23:10:28.548952603 (6h43m42.136008854s) [upload=6903]: Got next segment. Inline: false Aug 7 23:10:28.548991997 (6h43m42.136048238s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.548997387 (6h43m42.136053628s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:10:28.575505557 (6h43m42.162561808s) [seg_pos={0 0}, upload=6903]: Begin upload segment... Aug 7 23:10:28.575543988 (6h43m42.162600229s) [seg_pos={0 0}, upload=6903]: Done begin upload segment. Aug 7 23:10:28.575570117 (6h43m42.162626358s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.575623586 (6h43m42.162679827s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576079848 (6h43m42.163136089s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576336698 (6h43m42.163392938s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576464947 (6h43m42.163521189s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576739170 (6h43m42.16379541s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576930076 (6h43m42.163986316s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577101956 (6h43m42.164158207s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577333558 (6h43m42.164389799s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577523042 (6h43m42.164579293s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577694552 (6h43m42.164750793s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577950891 (6h43m42.165007132s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578056468 (6h43m42.165112719s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578412783 (6h43m42.165469065s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578497261 (6h43m42.165553512s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578896225 (6h43m42.165952466s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.726482559 (6h43m42.3135388s) [upload=6903]: Next returned nil segment Aug 7 23:10:28.726499210 (6h43m42.313555452s) [upload=6903]: Waiting for error group managing segments... Aug 7 23:10:28.733585057 (6h43m42.320641298s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.734493031 (6h43m42.321549272s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735107198 (6h43m42.322163439s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735334542 (6h43m42.322390784s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735677101 (6h43m42.322733343s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736304363 (6h43m42.323360613s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736733653 (6h43m42.323789894s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.737010621 (6h43m42.324066872s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.738308202 (6h43m42.325364454s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.739491822 (6h43m42.326548064s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.740628304 (6h43m42.327684545s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.741118448 (6h43m42.328174699s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743301704 (6h43m42.330357955s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743318215 (6h43m42.330374466s) [seg_pos={0 0}, upload=6903]: Segment reached optimal threshold of 13 pieces. Aug 7 23:10:28.743478223 (6h43m42.330534464s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743579422 (6h43m42.330635673s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Aug 7 23:10:28.743685970 (6h43m42.330742212s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:64866->104.36.82.15:4257: use of closed network connection; context canceled Aug 7 23:10:28.743721447 (6h43m42.330777688s) [seg_pos={0 0}, upload=6903]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Aug 7 23:10:28.743791097 (6h43m42.330847338s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:28.743796517 (6h43m42.330852758s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.784620930 (6h43m42.371677172s) [upload=6903]: Done uploading segments... Aug 7 23:10:28.784637271 (6h43m42.371693512s) [upload=6903]: Upload finished. err: <nil> Aug 7 23:11:14.178732655 (6h44m27.765788876s) [upload=6917]: Starting upload Aug 7 23:11:14.178789762 (6h44m27.765845983s) [upload=6917]: Done starting upload Aug 7 23:11:14.178845816 (6h44m27.765902038s) [upload=6917]: Uploading segments... Aug 7 23:11:14.179029189 (6h44m27.76608541s) [upload=6917]: Got next segment. Inline: false Aug 7 23:11:14.179063893 (6h44m27.766120104s) [upload=6917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:14.179069093 (6h44m27.766125314s) [upload=6917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:44.165125191 (6h44m57.752181432s) [upload=6917]: Next segment err: context deadline exceeded Aug 7 23:11:44.165321557 (6h44m57.752377798s) [upload=6917]: Done uploading segments... Aug 7 23:11:44.165369758 (6h44m57.752425998s) [upload=6917]: Upload finished. err: context deadline exceeded Duration: 778.190788ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002457529s FAIL (2025-08-07 23:13:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00247904s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X..X.......................................................X] duration: 1m0.000679923s FAIL (2025-08-07 23:13:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000701032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000592747s FAIL (2025-08-07 23:13:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 067983284s) [upload=6851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:39.538593443 (6h40m53.125649684s) [upload=6851]: Done uploading segments... Aug 7 23:07:39.538621665 (6h40m53.125677906s) [upload=6851]: Upload finished. err: <nil> Aug 7 23:08:26.599295831 (6h41m40.186352072s) [upload=6863]: Starting upload Aug 7 23:08:26.599379427 (6h41m40.186435668s) [upload=6863]: Done starting upload Aug 7 23:08:26.599448597 (6h41m40.186504838s) [upload=6863]: Uploading segments... Aug 7 23:08:26.599531050 (6h41m40.186587292s) [upload=6863]: Got next segment. Inline: true Aug 7 23:08:26.599538975 (6h41m40.186595216s) [upload=6863]: Waiting for error group managing segments... Aug 7 23:08:26.599587325 (6h41m40.186643566s) [upload=6863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:26.599593266 (6h41m40.186649517s) [upload=6863]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:26.599596923 (6h41m40.186653174s) [upload=6863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:26.662838039 (6h41m40.249894279s) [upload=6863]: Done uploading segments... Aug 7 23:08:26.662853056 (6h41m40.249909298s) [upload=6863]: Upload finished. err: <nil> Aug 7 23:09:07.080522893 (6h42m20.667579114s) [upload=6875]: Starting upload Aug 7 23:09:07.080579920 (6h42m20.667636141s) [upload=6875]: Done starting upload Aug 7 23:09:07.080644431 (6h42m20.667700642s) [upload=6875]: Uploading segments... Aug 7 23:09:07.080682552 (6h42m20.667738773s) [upload=6875]: Got next segment. Inline: true Aug 7 23:09:07.080688112 (6h42m20.667744334s) [upload=6875]: Waiting for error group managing segments... Aug 7 23:09:07.080725373 (6h42m20.667781594s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:07.080729751 (6h42m20.667785972s) [upload=6875]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:07.080732115 (6h42m20.667788326s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:07.161521184 (6h42m20.748577405s) [upload=6875]: Done uploading segments... Aug 7 23:09:07.161537344 (6h42m20.748593565s) [upload=6875]: Upload finished. err: <nil> Aug 7 23:09:32.587106585 (6h42m46.174162806s) [upload=6885]: Starting upload Aug 7 23:09:32.587221680 (6h42m46.174277901s) [upload=6885]: Done starting upload Aug 7 23:09:32.587315766 (6h42m46.174371977s) [upload=6885]: Uploading segments... Aug 7 23:09:32.587429769 (6h42m46.174486019s) [upload=6885]: Got next segment. Inline: true Aug 7 23:09:32.587436501 (6h42m46.174492722s) [upload=6885]: Waiting for error group managing segments... Aug 7 23:09:32.587485553 (6h42m46.174541764s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:32.587491184 (6h42m46.174547405s) [upload=6885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:32.587495031 (6h42m46.174551252s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.000609698s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 11.564907ms target: https://us1.storj.io/login FAIL (2025-08-07 23:13:47 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 11.591958ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 175.990782ms FAIL (2025-08-07 23:13:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 176.009017ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.743187405s FAIL (2025-08-07 23:13:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 1.743214945s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 24.785175ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:13:34 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 24.807986ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX.X......................................................XX] duration: 30.003647822s FAIL (2025-08-07 23:13:30 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.003674041s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.........................................................X] duration: 30.000443061s FAIL (2025-08-07 23:13:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000461155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 366.423263ms FAIL (2025-08-07 23:13:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== h43m23.967460908s) [upload=6897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:10.380410569 (6h43m23.96746681s) [upload=6897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:10.423498627 (6h43m24.010554838s) [upload=6897]: Done uploading segments... Aug 7 23:10:10.423508945 (6h43m24.010565157s) [upload=6897]: Upload finished. err: <nil> Aug 7 23:10:28.548503936 (6h43m42.135560176s) [upload=6903]: Starting upload Aug 7 23:10:28.548587942 (6h43m42.135644183s) [upload=6903]: Done starting upload Aug 7 23:10:28.548687107 (6h43m42.135743348s) [upload=6903]: Uploading segments... Aug 7 23:10:28.548952603 (6h43m42.136008854s) [upload=6903]: Got next segment. Inline: false Aug 7 23:10:28.548991997 (6h43m42.136048238s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.548997387 (6h43m42.136053628s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:10:28.575505557 (6h43m42.162561808s) [seg_pos={0 0}, upload=6903]: Begin upload segment... Aug 7 23:10:28.575543988 (6h43m42.162600229s) [seg_pos={0 0}, upload=6903]: Done begin upload segment. Aug 7 23:10:28.575570117 (6h43m42.162626358s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.575623586 (6h43m42.162679827s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576079848 (6h43m42.163136089s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576336698 (6h43m42.163392938s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576464947 (6h43m42.163521189s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576739170 (6h43m42.16379541s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.576930076 (6h43m42.163986316s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577101956 (6h43m42.164158207s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577333558 (6h43m42.164389799s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577523042 (6h43m42.164579293s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577694552 (6h43m42.164750793s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.577950891 (6h43m42.165007132s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578056468 (6h43m42.165112719s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578412783 (6h43m42.165469065s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578497261 (6h43m42.165553512s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.578896225 (6h43m42.165952466s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Uploading piece... Aug 7 23:10:28.726482559 (6h43m42.3135388s) [upload=6903]: Next returned nil segment Aug 7 23:10:28.726499210 (6h43m42.313555452s) [upload=6903]: Waiting for error group managing segments... Aug 7 23:10:28.733585057 (6h43m42.320641298s) [noise=true , address=45.91.116.9:4263 , piece_id=23ZBS3NX7IJ4QETO2ZNNVTZZRFXASARGSJC3GQTYPNJZIB673Y5A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.734493031 (6h43m42.321549272s) [noise=true , address=45.91.116.4:4940 , piece_id=MPWRCS54USFUITJZDQHCDXWB6X23WXVZ5L5DH3QRZKCQIGKZGCFA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735107198 (6h43m42.322163439s) [noise=true , address=45.91.116.10:4082 , piece_id=OMIBNZ2HNZYNIFFMA6JRC43ERJL4Q5GTDA2VSXAYK5MHUXAEJNTQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735334542 (6h43m42.322390784s) [noise=true , address=45.91.116.8:4047 , piece_id=RZMJRKPWKSYC2GBZOFRH2V35I4MFZI4O4YCU42NCRNKXXWFF3I3Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.735677101 (6h43m42.322733343s) [noise=true , address=45.91.116.14:4190 , piece_id=25IQPVVJCMEGBD54U7SBR4B2DF3SHMVMNNCPQ5BCSRZHR6ZCBZBA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736304363 (6h43m42.323360613s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZFFCHNDBQNCBC7LNSYTHN2B4NH3O3NMOPY6GGAWUMN3FJEJMQSA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.736733653 (6h43m42.323789894s) [noise=true , address=45.91.116.6:4975 , piece_id=BS3KPKXAUBPJ2QKDWVSIKXHDXA2B3SCHMQONC47UXYP3ZMKDDOZQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.737010621 (6h43m42.324066872s) [noise=true , address=77.90.31.12:4592 , piece_id=TRJXNNTS2CA64YE6MARCSCM5ONAYU6JM756GRJDO22GQC43DWJ4Q, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.738308202 (6h43m42.325364454s) [noise=true , address=45.91.116.4:4938 , piece_id=VKQGCPFCPJ4C2XFKFPBCHVH5SLUWGDF54GYET5V6AE6Z7M5ZLONA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.739491822 (6h43m42.326548064s) [noise=true , address=104.36.82.7:4912 , piece_id=EBWCUT7Z6ZFLRWCQH34MTVRGLFKZ4SFPUBGDQK4OFQ4GFSWTSBHA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.740628304 (6h43m42.327684545s) [noise=true , address=104.36.82.14:4223 , piece_id=ZEW6AY6ZAAZ6FNYWBAUAPD3CGFPBPER4SJXCEYOZH2HPLWIY5A6A, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.741118448 (6h43m42.328174699s) [noise=true , address=77.90.31.19:4842 , piece_id=P3VUJZWLUVXOC67C3PHKNY3KVDPAFJ65HBEMUOQD7ZBYPYSIP2QQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743301704 (6h43m42.330357955s) [noise=true , address=45.91.116.5:4297 , piece_id=L62H2ZU66BU744YEJEBI5DNSNKTUDWPKMWNXTDFGJQE2RWK6NTQA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743318215 (6h43m42.330374466s) [seg_pos={0 0}, upload=6903]: Segment reached optimal threshold of 13 pieces. Aug 7 23:10:28.743478223 (6h43m42.330534464s) [noise=true , address=45.91.116.14:4189 , piece_id=E2KVLTZ2QEQP3SDGFFUH4ABEWQDRMHLY3LQN4OUJTPTTLP4V7OHQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: <nil> Aug 7 23:10:28.743579422 (6h43m42.330635673s) [noise=true , address=104.36.82.6:4875 , piece_id=ELDGBZIUI2VPU6ZAQAO45IA7Y5CQSKHUDJTGOQGBKNBCW4ACYWRQ, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Aug 7 23:10:28.743685970 (6h43m42.330742212s) [noise=true , address=104.36.82.15:4257 , piece_id=JZTCXQPUTQPLBCF3Q3HEEMCJ4K2R2L2XUNKHH3IJPNSIGWAOI5AA, seg_pos={0 0}, upload=6903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:64866->104.36.82.15:4257: use of closed network connection; context canceled Aug 7 23:10:28.743721447 (6h43m42.330777688s) [seg_pos={0 0}, upload=6903]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Aug 7 23:10:28.743791097 (6h43m42.330847338s) [upload=6903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:10:28.743796517 (6h43m42.330852758s) [upload=6903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.784620930 (6h43m42.371677172s) [upload=6903]: Done uploading segments... Aug 7 23:10:28.784637271 (6h43m42.371693512s) [upload=6903]: Upload finished. err: <nil> Aug 7 23:11:14.178732655 (6h44m27.765788876s) [upload=6917]: Starting upload Aug 7 23:11:14.178789762 (6h44m27.765845983s) [upload=6917]: Done starting upload Aug 7 23:11:14.178845816 (6h44m27.765902038s) [upload=6917]: Uploading segments... Aug 7 23:11:14.179029189 (6h44m27.76608541s) [upload=6917]: Got next segment. Inline: false Aug 7 23:11:14.179063893 (6h44m27.766120104s) [upload=6917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:14.179069093 (6h44m27.766125314s) [upload=6917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:44.165125191 (6h44m57.752181432s) [upload=6917]: Next segment err: context deadline exceeded Aug 7 23:11:44.165321557 (6h44m57.752377798s) [upload=6917]: Done uploading segments... Aug 7 23:11:44.165369758 (6h44m57.752425998s) [upload=6917]: Upload finished. err: context deadline exceeded Duration: 366.446827ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00135095s FAIL (2025-08-07 23:13:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 23:07:04.580716557 (6h40m18.167772798s) [upload=6842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:04.645715104 (6h40m18.232771345s) [upload=6842]: Done uploading segments... Aug 7 23:07:04.645757051 (6h40m18.232813293s) [upload=6842]: Upload finished. err: <nil> Aug 7 23:07:39.480764649 (6h40m53.067820861s) [upload=6851]: Starting upload Aug 7 23:07:39.480817980 (6h40m53.067874191s) [upload=6851]: Done starting upload Aug 7 23:07:39.480862783 (6h40m53.067918994s) [upload=6851]: Uploading segments... Aug 7 23:07:39.480888400 (6h40m53.067944612s) [upload=6851]: Got next segment. Inline: true Aug 7 23:07:39.480891927 (6h40m53.067948138s) [upload=6851]: Waiting for error group managing segments... Aug 7 23:07:39.480921543 (6h40m53.067977754s) [upload=6851]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:39.480925130 (6h40m53.06798134s) [upload=6851]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:39.480927073 (6h40m53.067983284s) [upload=6851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:39.538593443 (6h40m53.125649684s) [upload=6851]: Done uploading segments... Aug 7 23:07:39.538621665 (6h40m53.125677906s) [upload=6851]: Upload finished. err: <nil> Aug 7 23:08:26.599295831 (6h41m40.186352072s) [upload=6863]: Starting upload Aug 7 23:08:26.599379427 (6h41m40.186435668s) [upload=6863]: Done starting upload Aug 7 23:08:26.599448597 (6h41m40.186504838s) [upload=6863]: Uploading segments... Aug 7 23:08:26.599531050 (6h41m40.186587292s) [upload=6863]: Got next segment. Inline: true Aug 7 23:08:26.599538975 (6h41m40.186595216s) [upload=6863]: Waiting for error group managing segments... Aug 7 23:08:26.599587325 (6h41m40.186643566s) [upload=6863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:26.599593266 (6h41m40.186649517s) [upload=6863]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:26.599596923 (6h41m40.186653174s) [upload=6863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:26.662838039 (6h41m40.249894279s) [upload=6863]: Done uploading segments... Aug 7 23:08:26.662853056 (6h41m40.249909298s) [upload=6863]: Upload finished. err: <nil> Aug 7 23:09:07.080522893 (6h42m20.667579114s) [upload=6875]: Starting upload Aug 7 23:09:07.080579920 (6h42m20.667636141s) [upload=6875]: Done starting upload Aug 7 23:09:07.080644431 (6h42m20.667700642s) [upload=6875]: Uploading segments... Aug 7 23:09:07.080682552 (6h42m20.667738773s) [upload=6875]: Got next segment. Inline: true Aug 7 23:09:07.080688112 (6h42m20.667744334s) [upload=6875]: Waiting for error group managing segments... Aug 7 23:09:07.080725373 (6h42m20.667781594s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:07.080729751 (6h42m20.667785972s) [upload=6875]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:07.080732115 (6h42m20.667788326s) [upload=6875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:07.161521184 (6h42m20.748577405s) [upload=6875]: Done uploading segments... Aug 7 23:09:07.161537344 (6h42m20.748593565s) [upload=6875]: Upload finished. err: <nil> Aug 7 23:09:32.587106585 (6h42m46.174162806s) [upload=6885]: Starting upload Aug 7 23:09:32.587221680 (6h42m46.174277901s) [upload=6885]: Done starting upload Aug 7 23:09:32.587315766 (6h42m46.174371977s) [upload=6885]: Uploading segments... Aug 7 23:09:32.587429769 (6h42m46.174486019s) [upload=6885]: Got next segment. Inline: true Aug 7 23:09:32.587436501 (6h42m46.174492722s) [upload=6885]: Waiting for error group managing segments... Aug 7 23:09:32.587485553 (6h42m46.174541764s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:32.587491184 (6h42m46.174547405s) [upload=6885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:32.587495031 (6h42m46.174551252s) [upload=6885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:32.637317298 (6h42m46.224373519s) [upload=6885]: Done uploading segments... Aug 7 23:09:32.637336604 (6h42m46.224392815s) [upload=6885]: Upload finished. err: <nil> Aug 7 23:10:01.302037808 (6h43m14.889094029s) [upload=6894]: Starting upload Aug 7 23:10:01.302081329 (6h43m14.88913755s) [upload=6894]: Done starting upload Aug 7 23:10:01.302122596 (6h43m14.889178817s) [upload=6894]: Uploading segments... Aug 7 23:10:01.302148996 (6h43m14.889205207s) [upload=6894]: Got next segment. Inline: true Aug 7 23:10:01.302152572 (6h43m14.889208793s) [upload=6894]: Waiting for error group managing segments... Aug 7 23:10:01.302181606 (6h43m14.889237817s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:01.302185193 (6h43m14.889241414s) [upload=6894]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:01.302188138 (6h43m14.889244359s) [upload=6894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:01.364607810 (6h43m14.951664141s) [upload=6894]: Done uploading segments... Aug 7 23:10:01.364634490 (6h43m14.951690731s) [upload=6894]: Upload finished. err: <nil> Aug 7 23:10:19.460675504 (6h43m33.047731715s) [upload=6901]: Starting upload Aug 7 23:10:19.460729253 (6h43m33.047785474s) [upload=6901]: Done starting upload Aug 7 23:10:19.460773937 (6h43m33.047830158s) [upload=6901]: Uploading segments... Aug 7 23:10:19.460814664 (6h43m33.047870885s) [upload=6901]: Got next segment. Inline: true Aug 7 23:10:19.460818461 (6h43m33.047874672s) [upload=6901]: Waiting for error group managing segments... Aug 7 23:10:19.460851652 (6h43m33.047907863s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:19.460855149 (6h43m33.04791136s) [upload=6901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:19.460857593 (6h43m33.047913814s) [upload=6901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:19.553699402 (6h43m33.140755623s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001378481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000952103s FAIL (2025-08-07 23:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gmentMakeInline Aug 7 23:04:47.565439458 (6h38m1.152495679s) [upload=6795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:04:47.635627853 (6h38m1.222684103s) [upload=6795]: Done uploading segments... Aug 7 23:04:47.635648982 (6h38m1.222705234s) [upload=6795]: Upload finished. err: <nil> Aug 7 23:05:53.001092736 (6h39m6.588148957s) [upload=6817]: Starting upload Aug 7 23:05:53.001148301 (6h39m6.588204522s) [upload=6817]: Done starting upload Aug 7 23:05:53.001225214 (6h39m6.588281445s) [upload=6817]: Uploading segments... Aug 7 23:05:53.001266261 (6h39m6.588322482s) [upload=6817]: Got next segment. Inline: true Aug 7 23:05:53.001273254 (6h39m6.588329465s) [upload=6817]: Waiting for error group managing segments... Aug 7 23:05:53.001323577 (6h39m6.588379798s) [upload=6817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:53.001330220 (6h39m6.588386441s) [upload=6817]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:53.001332965 (6h39m6.588389176s) [upload=6817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:53.059652995 (6h39m6.646709206s) [upload=6817]: Done uploading segments... Aug 7 23:05:53.059668294 (6h39m6.646724515s) [upload=6817]: Upload finished. err: <nil> Aug 7 23:06:48.439632034 (6h40m2.026688255s) [upload=6833]: Starting upload Aug 7 23:06:48.439692607 (6h40m2.026748818s) [upload=6833]: Done starting upload Aug 7 23:06:48.439737180 (6h40m2.026793391s) [upload=6833]: Uploading segments... Aug 7 23:06:48.439763910 (6h40m2.026820122s) [upload=6833]: Got next segment. Inline: true Aug 7 23:06:48.439767437 (6h40m2.026823658s) [upload=6833]: Waiting for error group managing segments... Aug 7 23:06:48.439799687 (6h40m2.026855908s) [upload=6833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:48.439803343 (6h40m2.026859564s) [upload=6833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:48.439805758 (6h40m2.026861979s) [upload=6833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:48.578044013 (6h40m2.165100234s) [upload=6833]: Done uploading segments... Aug 7 23:06:48.578050885 (6h40m2.165107106s) [upload=6833]: Upload finished. err: <nil> Aug 7 23:07:43.592700568 (6h40m57.179756809s) [upload=6854]: Starting upload Aug 7 23:07:43.592773794 (6h40m57.179830045s) [upload=6854]: Done starting upload Aug 7 23:07:43.592853843 (6h40m57.179910084s) [upload=6854]: Uploading segments... Aug 7 23:07:43.592922552 (6h40m57.179978813s) [upload=6854]: Got next segment. Inline: true Aug 7 23:07:43.592928533 (6h40m57.179984773s) [upload=6854]: Waiting for error group managing segments... Aug 7 23:07:43.592972054 (6h40m57.180028305s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:43.592977975 (6h40m57.180034216s) [upload=6854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:43.593034962 (6h40m57.180091213s) [upload=6854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:43.649755305 (6h40m57.236811607s) [upload=6854]: Done uploading segments... Aug 7 23:07:43.649779701 (6h40m57.236835953s) [upload=6854]: Upload finished. err: <nil> Aug 7 23:08:56.878235045 (6h42m10.465291266s) [upload=6873]: Starting upload Aug 7 23:08:56.878316127 (6h42m10.465372348s) [upload=6873]: Done starting upload Aug 7 23:08:56.878378183 (6h42m10.465434404s) [upload=6873]: Uploading segments... Aug 7 23:08:56.878424890 (6h42m10.46548114s) [upload=6873]: Got next segment. Inline: true Aug 7 23:08:56.878430340 (6h42m10.465486561s) [upload=6873]: Waiting for error group managing segments... Aug 7 23:08:56.878474843 (6h42m10.465531064s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:56.878479702 (6h42m10.465535923s) [upload=6873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:56.878483188 (6h42m10.465539409s) [upload=6873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:56.941171742 (6h42m10.528227963s) [upload=6873]: Done uploading segments... Aug 7 23:08:56.941188142 (6h42m10.528244354s) [upload=6873]: Upload finished. err: <nil> Aug 7 23:09:42.944491406 (6h42m56.531547628s) [upload=6888]: Starting upload Aug 7 23:09:42.944574772 (6h42m56.531630993s) [upload=6888]: Done starting upload Aug 7 23:09:42.944630365 (6h42m56.531686576s) [upload=6888]: Uploading segments... Aug 7 23:09:42.944668176 (6h42m56.531724397s) [upload=6888]: Got next segment. Inline: true Aug 7 23:09:42.944673766 (6h42m56.531729977s) [upload=6888]: Waiting for error group managing segments... Aug 7 23:09:42.944715465 (6h42m56.531771686s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:42.944720784 (6h42m56.531777005s) [upload=6888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:42.944724171 (6h42m56.531780392s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:43.008585323 (6h42m56.595641534s) [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000971128s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)