Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.001115025s FAIL (2025-08-07 21:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001148397s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [..............................XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 36.971170635s FAIL (2025-08-07 21:42:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.971182949s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X...X...X......] duration: 109.754661ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:42:32 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 109.771995ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration: 30.00042147s FAIL (2025-08-07 21:42:31 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7 (5h14m28.062269058s) [upload=5388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:25.266685321 (5h14m38.853741562s) [upload=5388]: Done uploading segments... Aug 7 21:41:25.266706391 (5h14m38.853762631s) [upload=5388]: Upload finished. err: <nil> Aug 7 21:41:41.070176076 (5h14m54.657232316s) [upload=5396]: Starting upload Aug 7 21:41:41.070246557 (5h14m54.657302798s) [upload=5396]: Done starting upload Aug 7 21:41:41.070459143 (5h14m54.657515385s) [upload=5396]: Uploading segments... Aug 7 21:41:41.070697238 (5h14m54.657753479s) [upload=5396]: Got next segment. Inline: false Aug 7 21:41:41.070732264 (5h14m54.657788505s) [upload=5396]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:41.070737293 (5h14m54.657793524s) [upload=5396]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:41.825834701 (5h14m55.412890922s) [seg_pos={0 0}, upload=5396]: Begin upload segment... Aug 7 21:41:41.825865027 (5h14m55.412921237s) [seg_pos={0 0}, upload=5396]: Done begin upload segment. Aug 7 21:41:41.825886015 (5h14m55.412942237s) [noise=true , address=77.90.31.12:4592 , piece_id=75OFHQZRJKZYL3QJ37T6QLDED663WXIEFAHTH2BQZLZ46PY5YPLQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.825988818 (5h14m55.413045068s) [noise=true , address=45.91.116.5:4298 , piece_id=C3BQRSMSRSM4FJ7SWEBPHH7UKNUJWCDD5AEEBEAYN6OMCM2VRSWA, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826203979 (5h14m55.413260201s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNKX2DGOGITTOYV7IH73WDLAPXHXWG5V5WFIJDTM5RUJQYP6MLQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826435742 (5h14m55.413491953s) [noise=true , address=77.90.31.13:4627 , piece_id=7MG24AD6AUVL6X66RZJ43OSAM7FLBB33KQY4ENE4Q3NR6PA63YQQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826541530 (5h14m55.41359777s) [noise=true , address=45.91.116.5:4299 , piece_id=XY3RW62XNQVLM6NTAZTBF26HWTY5KR3ZEJGVYTSTENY2B77YLBMA, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826653037 (5h14m55.413709248s) [noise=true , address=77.90.31.14:4661 , piece_id=U3YYDMWVGG4NEV5OSK4ATJ2RNTEYCIXKH3KXY7UR3RGJXEUA25IA, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826875332 (5h14m55.413931543s) [noise=true , address=77.90.31.10:4519 , piece_id=OQZPQPV2NUVKLMSHONEMH2HUK52KXL2E6NXXSO2J6CP7IDTRVT7Q, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.826921499 (5h14m55.41397774s) [noise=true , address=104.36.82.11:4114 , piece_id=KB2SDFXXBJDEYUHUNIO3YSANNDXVREF46SGOSWQ3YBHRR42TFD4A, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827091606 (5h14m55.414147827s) [noise=true , address=45.91.116.4:4940 , piece_id=ETERKOUZIHEOKPCAJPSUBGXFMDLKEELCVMIW34LR7G66FBFMUNTA, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827323819 (5h14m55.41438003s) [noise=true , address=77.90.31.17:4771 , piece_id=IYUYS3YHOD44ZF2OGKQX7JCFVW2GBU5S2TVC3FZ2M3JNF67TP4CQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827393500 (5h14m55.414449742s) [noise=true , address=104.36.82.6:4875 , piece_id=B263E3OQZQDAAYHN3YIPJ4RBLTNOFNOXAZWYTVEGGVRM2YES3PBQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827542938 (5h14m55.414599159s) [noise=true , address=104.36.82.7:4911 , piece_id=GLWOEWKACQVRHXQ6EY45IYR5N7VCP75FUZIYT5WRQG2VXULRUIUA, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827778078 (5h14m55.414834318s) [noise=true , address=104.36.82.10:4077 , piece_id=LZBNWOC4E35SX6FKBLATK2QFZDUWA7CBA5RY422YHVMMKO5XDSBQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.827801491 (5h14m55.414857712s) [noise=true , address=104.36.82.5:4837 , piece_id=32VX7X2CPAQKJGKQYNDUF2U6GPAWZKEWYALRTRSFC6TPIQJBBXCQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.828365735 (5h14m55.415421976s) [noise=true , address=45.91.116.8:4048 , piece_id=XZNUB5SDPWSYVGHGMFFQZ4BZMBP2GMKF76GXWQ2MXAGT42I6I4FQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.828913136 (5h14m55.415969378s) [noise=true , address=104.36.82.2:4732 , piece_id=2HK2G5XCNE6UN3COUOG6V4IVCLTRVEMNNCQMV5SZLXRFTO2FJUYQ, seg_pos={0 0}, upload=5396]: Uploading piece... Aug 7 21:41:41.934607129 (5h14m55.52166334s) [upload=5396]: Next returned nil segment Aug 7 21:41:41.934619282 (5h14m55.521675503s) [upload=5396]: Waiting for error group managing segments... Aug 7 21:41:41.945118551 (5h14m55.532174772s) [noise=true , address=45.91.116.4:4940 , piece_id=ETERKOUZIHEOKPCAJPSUBGXFMDLKEELCVMIW34LR7G66FBFMUNTA, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.945901252 (5h14m55.532957473s) [noise=true , address=45.91.116.8:4048 , piece_id=XZNUB5SDPWSYVGHGMFFQZ4BZMBP2GMKF76GXWQ2MXAGT42I6I4FQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.946161258 (5h14m55.533217498s) [noise=true , address=45.91.116.5:4298 , piece_id=C3BQRSMSRSM4FJ7SWEBPHH7UKNUJWCDD5AEEBEAYN6OMCM2VRSWA, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.946386648 (5h14m55.533442869s) [noise=true , address=77.90.31.12:4592 , piece_id=75OFHQZRJKZYL3QJ37T6QLDED663WXIEFAHTH2BQZLZ46PY5YPLQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.949024992 (5h14m55.536081243s) [noise=true , address=45.91.116.5:4299 , piece_id=XY3RW62XNQVLM6NTAZTBF26HWTY5KR3ZEJGVYTSTENY2B77YLBMA, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.949218724 (5h14m55.536274945s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNKX2DGOGITTOYV7IH73WDLAPXHXWG5V5WFIJDTM5RUJQYP6MLQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.949657033 (5h14m55.536713244s) [noise=true , address=77.90.31.13:4627 , piece_id=7MG24AD6AUVL6X66RZJ43OSAM7FLBB33KQY4ENE4Q3NR6PA63YQQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.949683371 (5h14m55.536739612s) [noise=true , address=104.36.82.2:4732 , piece_id=2HK2G5XCNE6UN3COUOG6V4IVCLTRVEMNNCQMV5SZLXRFTO2FJUYQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.950070354 (5h14m55.537126575s) [noise=true , address=104.36.82.11:4114 , piece_id=KB2SDFXXBJDEYUHUNIO3YSANNDXVREF46SGOSWQ3YBHRR42TFD4A, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.953623165 (5h14m55.540679407s) [noise=true , address=104.36.82.6:4875 , piece_id=B263E3OQZQDAAYHN3YIPJ4RBLTNOFNOXAZWYTVEGGVRM2YES3PBQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.957708261 (5h14m55.544764482s) [noise=true , address=104.36.82.5:4837 , piece_id=32VX7X2CPAQKJGKQYNDUF2U6GPAWZKEWYALRTRSFC6TPIQJBBXCQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.964243811 (5h14m55.551300021s) [noise=true , address=77.90.31.10:4519 , piece_id=OQZPQPV2NUVKLMSHONEMH2HUK52KXL2E6NXXSO2J6CP7IDTRVT7Q, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.964842018 (5h14m55.551898239s) [noise=true , address=77.90.31.14:4661 , piece_id=U3YYDMWVGG4NEV5OSK4ATJ2RNTEYCIXKH3KXY7UR3RGJXEUA25IA, seg_pos={0 0}, upload=5396]: Done uploading piece. err: <nil> Aug 7 21:41:41.964860783 (5h14m55.551917004s) [seg_pos={0 0}, upload=5396]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:41.964962292 (5h14m55.552018503s) [noise=true , address=77.90.31.17:4771 , piece_id=IYUYS3YHOD44ZF2OGKQX7JCFVW2GBU5S2TVC3FZ2M3JNF67TP4CQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:41.965019739 (5h14m55.55207595s) [noise=true , address=104.36.82.10:4077 , piece_id=LZBNWOC4E35SX6FKBLATK2QFZDUWA7CBA5RY422YHVMMKO5XDSBQ, seg_pos={0 0}, upload=5396]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:41.965087696 (5h14m55.552143947s) [noise=true , address=104.36.82.7:4911 , piece_id=GLWOEWKACQVRHXQ6EY45IYR5N7VCP75FUZIYT5WRQG2VXULRUIUA, seg_pos={0 0}, upload=5396]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:41.965118783 (5h14m55.552175005s) [seg_pos={0 0}, upload=5396]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:41.965239138 (5h14m55.552295349s) [upload=5396]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:41.965243687 (5h14m55.552299898s) [upload=5396]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:46.509965896 (5h15m0.097022118s) [upload=5396]: Done uploading segments... Aug 7 21:41:46.510002955 (5h15m0.097059167s) [upload=5396]: Upload finished. err: <nil> Aug 7 21:42:01.196715658 (5h15m14.783771899s) [upload=5401]: Starting upload Aug 7 21:42:01.196842946 (5h15m14.783899187s) [upload=5401]: Done starting upload Aug 7 21:42:01.196915852 (5h15m14.783972103s) [upload=5401]: Uploading segments... Aug 7 21:42:01.197283067 (5h15m14.784339309s) [upload=5401]: Got next segment. Inline: false Aug 7 21:42:01.197355652 (5h15m14.784411984s) [upload=5401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:01.197365230 (5h15m14.784421472s) [upload=5401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:31.170606605 (5h15m44.757662826s) [upload=5401]: Next segment err: context deadline exceeded Aug 7 21:42:31.170727651 (5h15m44.757783872s) [upload=5401]: Done uploading segments... Aug 7 21:42:31.170749551 (5h15m44.757805773s) [upload=5401]: Upload finished. err: context deadline exceeded Duration:* 30.000440086s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [................................X........XX...X.............] duration: 15.587109923s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:42:31 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 15.58713558s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.195724858s FAIL (2025-08-07 21:42:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 3.195750598s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 211.31973ms FAIL (2025-08-07 21:42:18 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:26.513648977 (5h11m40.100705218s) [upload=5344]: Starting upload Aug 7 21:38:26.513731310 (5h11m40.100787552s) [upload=5344]: Done starting upload Aug 7 21:38:26.513792245 (5h11m40.100848496s) [upload=5344]: Uploading segments... Aug 7 21:38:26.513834112 (5h11m40.100890354s) [upload=5344]: Got next segment. Inline: true Aug 7 21:38:26.513839653 (5h11m40.100895904s) [upload=5344]: Waiting for error group managing segments... Aug 7 21:38:26.513884006 (5h11m40.100940247s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:26.513889947 (5h11m40.100946188s) [upload=5344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:26.513893434 (5h11m40.100949675s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:56.502052245 (5h12m10.089108537s) [upload=5344]: Done uploading segments... Aug 7 21:38:56.502102730 (5h12m10.089158972s) [upload=5344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:24.008006172 (5h12m37.595062394s) [upload=5359]: Starting upload Aug 7 21:39:24.008059211 (5h12m37.595115422s) [upload=5359]: Done starting upload Aug 7 21:39:24.008103604 (5h12m37.595159835s) [upload=5359]: Uploading segments... Aug 7 21:39:24.008156191 (5h12m37.595212413s) [upload=5359]: Got next segment. Inline: true Aug 7 21:39:24.008162193 (5h12m37.595218414s) [upload=5359]: Waiting for error group managing segments... Aug 7 21:39:24.008207327 (5h12m37.595263548s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:24.008212668 (5h12m37.595268878s) [upload=5359]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:24.008216064 (5h12m37.595272285s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:53.997226503 (5h13m7.584282744s) [upload=5359]: Done uploading segments... Aug 7 21:39:53.997270334 (5h13m7.584326586s) [upload=5359]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:36.779619947 (5h13m50.366676188s) [upload=5383]: Starting upload Aug 7 21:40:36.779696139 (5h13m50.36675238s) [upload=5383]: Done starting upload Aug 7 21:40:36.779752775 (5h13m50.366809016s) [upload=5383]: Uploading segments... Aug 7 21:40:36.779788803 (5h13m50.366845044s) [upload=5383]: Got next segment. Inline: true Aug 7 21:40:36.779794423 (5h13m50.366850655s) [upload=5383]: Waiting for error group managing segments... Aug 7 21:40:36.779835950 (5h13m50.366892191s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:36.779841010 (5h13m50.366897241s) [upload=5383]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:36.779844195 (5h13m50.366900437s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:06.765292852 (5h14m20.352349064s) [upload=5383]: Done uploading segments... Aug 7 21:41:06.765314052 (5h14m20.352370273s) [upload=5383]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:47.961322510 (5h15m1.548378802s) [upload=5399]: Starting upload Aug 7 21:41:47.961442915 (5h15m1.548499166s) [upload=5399]: Done starting upload Aug 7 21:41:47.961510201 (5h15m1.548566443s) [upload=5399]: Uploading segments... Aug 7 21:41:47.961591062 (5h15m1.548647313s) [upload=5399]: Got next segment. Inline: true Aug 7 21:41:47.961599218 (5h15m1.548655458s) [upload=5399]: Waiting for error group managing segments... Aug 7 21:41:47.961647759 (5h15m1.548703999s) [upload=5399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:47.961653760 (5h15m1.548710001s) [upload=5399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:47.961657546 (5h15m1.548713787s) [upload=5399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:17.948677724 (5h15m31.535733945s) [upload=5399]: Done uploading segments... Aug 7 21:42:17.948691099 (5h15m31.53574731s) [upload=5399]: Upload finished. err: metaclient: context deadline exceeded Duration: 211.337002ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 39 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001431799s FAIL (2025-08-07 21:42:17 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:26.513648977 (5h11m40.100705218s) [upload=5344]: Starting upload Aug 7 21:38:26.513731310 (5h11m40.100787552s) [upload=5344]: Done starting upload Aug 7 21:38:26.513792245 (5h11m40.100848496s) [upload=5344]: Uploading segments... Aug 7 21:38:26.513834112 (5h11m40.100890354s) [upload=5344]: Got next segment. Inline: true Aug 7 21:38:26.513839653 (5h11m40.100895904s) [upload=5344]: Waiting for error group managing segments... Aug 7 21:38:26.513884006 (5h11m40.100940247s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:26.513889947 (5h11m40.100946188s) [upload=5344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:26.513893434 (5h11m40.100949675s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:56.502052245 (5h12m10.089108537s) [upload=5344]: Done uploading segments... Aug 7 21:38:56.502102730 (5h12m10.089158972s) [upload=5344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:24.008006172 (5h12m37.595062394s) [upload=5359]: Starting upload Aug 7 21:39:24.008059211 (5h12m37.595115422s) [upload=5359]: Done starting upload Aug 7 21:39:24.008103604 (5h12m37.595159835s) [upload=5359]: Uploading segments... Aug 7 21:39:24.008156191 (5h12m37.595212413s) [upload=5359]: Got next segment. Inline: true Aug 7 21:39:24.008162193 (5h12m37.595218414s) [upload=5359]: Waiting for error group managing segments... Aug 7 21:39:24.008207327 (5h12m37.595263548s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:24.008212668 (5h12m37.595268878s) [upload=5359]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:24.008216064 (5h12m37.595272285s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:53.997226503 (5h13m7.584282744s) [upload=5359]: Done uploading segments... Aug 7 21:39:53.997270334 (5h13m7.584326586s) [upload=5359]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:36.779619947 (5h13m50.366676188s) [upload=5383]: Starting upload Aug 7 21:40:36.779696139 (5h13m50.36675238s) [upload=5383]: Done starting upload Aug 7 21:40:36.779752775 (5h13m50.366809016s) [upload=5383]: Uploading segments... Aug 7 21:40:36.779788803 (5h13m50.366845044s) [upload=5383]: Got next segment. Inline: true Aug 7 21:40:36.779794423 (5h13m50.366850655s) [upload=5383]: Waiting for error group managing segments... Aug 7 21:40:36.779835950 (5h13m50.366892191s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:36.779841010 (5h13m50.366897241s) [upload=5383]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:36.779844195 (5h13m50.366900437s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:06.765292852 (5h14m20.352349064s) [upload=5383]: Done uploading segments... Aug 7 21:41:06.765314052 (5h14m20.352370273s) [upload=5383]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:47.961322510 (5h15m1.548378802s) [upload=5399]: Starting upload Aug 7 21:41:47.961442915 (5h15m1.548499166s) [upload=5399]: Done starting upload Aug 7 21:41:47.961510201 (5h15m1.548566443s) [upload=5399]: Uploading segments... Aug 7 21:41:47.961591062 (5h15m1.548647313s) [upload=5399]: Got next segment. Inline: true Aug 7 21:41:47.961599218 (5h15m1.548655458s) [upload=5399]: Waiting for error group managing segments... Aug 7 21:41:47.961647759 (5h15m1.548703999s) [upload=5399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:47.961653760 (5h15m1.548710001s) [upload=5399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:47.961657546 (5h15m1.548713787s) [upload=5399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:17.948677724 (5h15m31.535733945s) [upload=5399]: Done uploading segments... Aug 7 21:42:17.948691099 (5h15m31.53574731s) [upload=5399]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001453139s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 234.015172ms FAIL (2025-08-07 21:42:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 234.038426ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000458069s FAIL (2025-08-07 21:42:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:15.177436931 (5h10m28.764493172s) [upload=5324]: Starting upload Aug 7 21:37:15.177533852 (5h10m28.764590103s) [upload=5324]: Done starting upload Aug 7 21:37:15.177602810 (5h10m28.764659061s) [upload=5324]: Uploading segments... Aug 7 21:37:15.177645340 (5h10m28.764701581s) [upload=5324]: Got next segment. Inline: true Aug 7 21:37:15.177650961 (5h10m28.764707202s) [upload=5324]: Waiting for error group managing segments... Aug 7 21:37:15.177698649 (5h10m28.76475489s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:15.177704300 (5h10m28.764760541s) [upload=5324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:15.177708177 (5h10m28.764764428s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:40.730185715 (5h10m54.317241957s) [upload=5324]: Done uploading segments... Aug 7 21:37:40.730203639 (5h10m54.31725988s) [upload=5324]: Upload finished. err: <nil> Aug 7 21:38:15.287715124 (5h11m28.874771365s) [upload=5341]: Starting upload Aug 7 21:38:15.287780276 (5h11m28.874836516s) [upload=5341]: Done starting upload Aug 7 21:38:15.287844856 (5h11m28.874901107s) [upload=5341]: Uploading segments... Aug 7 21:38:15.287889961 (5h11m28.874946201s) [upload=5341]: Got next segment. Inline: true Aug 7 21:38:15.287895600 (5h11m28.874951842s) [upload=5341]: Waiting for error group managing segments... Aug 7 21:38:15.287984066 (5h11m28.875040317s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:15.287990077 (5h11m28.875046318s) [upload=5341]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:15.287993564 (5h11m28.875049804s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:15.275679218 (5h12m28.862735459s) [upload=5341]: Done uploading segments... Aug 7 21:39:15.275708622 (5h12m28.862764883s) [upload=5341]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:15.358605080 (5h12m28.945661322s) [upload=5358]: Starting upload Aug 7 21:39:15.358690641 (5h12m28.945746882s) [upload=5358]: Done starting upload Aug 7 21:39:15.358748869 (5h12m28.945805111s) [upload=5358]: Uploading segments... Aug 7 21:39:15.358790938 (5h12m28.94584718s) [upload=5358]: Got next segment. Inline: true Aug 7 21:39:15.358796779 (5h12m28.94585302s) [upload=5358]: Waiting for error group managing segments... Aug 7 21:39:15.358840480 (5h12m28.945896721s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:15.358846441 (5h12m28.945902683s) [upload=5358]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:15.358849918 (5h12m28.945906159s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:15.344303433 (5h13m28.931359654s) [upload=5358]: Done uploading segments... Aug 7 21:40:15.344323120 (5h13m28.931379331s) [upload=5358]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:15.438334176 (5h13m29.025390397s) [upload=5373]: Starting upload Aug 7 21:40:15.438379379 (5h13m29.025435601s) [upload=5373]: Done starting upload Aug 7 21:40:15.438418503 (5h13m29.025474714s) [upload=5373]: Uploading segments... Aug 7 21:40:15.438447206 (5h13m29.025503427s) [upload=5373]: Got next segment. Inline: true Aug 7 21:40:15.438450683 (5h13m29.025506894s) [upload=5373]: Waiting for error group managing segments... Aug 7 21:40:15.438481920 (5h13m29.025538141s) [upload=5373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:15.438487511 (5h13m29.025543722s) [upload=5373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:15.438489826 (5h13m29.025546047s) [upload=5373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:15.423387101 (5h14m29.010443342s) [upload=5373]: Done uploading segments... Aug 7 21:41:15.423408011 (5h14m29.010464251s) [upload=5373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:15.514709386 (5h14m29.101765628s) [upload=5390]: Starting upload Aug 7 21:41:15.514790870 (5h14m29.10184711s) [upload=5390]: Done starting upload Aug 7 21:41:15.514852845 (5h14m29.101909086s) [upload=5390]: Uploading segments... Aug 7 21:41:15.514892409 (5h14m29.10194865s) [upload=5390]: Got next segment. Inline: true Aug 7 21:41:15.514898240 (5h14m29.101954481s) [upload=5390]: Waiting for error group managing segments... Aug 7 21:41:15.514941951 (5h14m29.101998192s) [upload=5390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:15.514947551 (5h14m29.102003793s) [upload=5390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:15.514951138 (5h14m29.102007389s) [upload=5390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:15.497042181 (5h15m29.084098402s) [upload=5390]: Done uploading segments... Aug 7 21:42:15.497058301 (5h15m29.084114512s) [upload=5390]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000472817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXX.X] duration: 1m0.00152429s FAIL (2025-08-07 21:42:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g 7 21:40:46.022110693 (5h13m59.609166914s) [noise=true , address=104.36.82.44:4891 , piece_id=D33W4EHVQ26LIOJ2KG5RKUHRZSGXAIGS2PQSIURYKGMP63RBTBGA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.022369437 (5h13m59.609425648s) [noise=true , address=104.36.82.14:4228 , piece_id=USWUBYJMTZKFFN3DZIKQQD7PW6QAKBRUHJZRCBO4DXTHR2PQDHRQ, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.022373765 (5h13m59.609429986s) [noise=true , address=64.49.13.139:4471 , piece_id=2OSYONIOJUEZXMYFSTL7NRN7HG6KXJOHFUL7TIDW73C4VKYF3W3A, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.022584097 (5h13m59.609640338s) [noise=true , address=104.36.82.56:4261 , piece_id=Z27LNZSXSYSUMQXEW2DI3C54ZJK26ALV653EINMYNU6COTMAMPBA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.022796634 (5h13m59.609852855s) [noise=true , address=104.36.82.12:4172 , piece_id=2SN23SJESN46DLA25WJULJLVGE7QWLQW2QJFAUA3WUVICJ5CKLUA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.023027314 (5h13m59.610083535s) [noise=true , address=104.36.82.54:4183 , piece_id=ZOCQQFJKFATMPRR2PKHLTSWN2D3QXAMZ74K2A5KTKYQAHEGA2BUA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.023255801 (5h13m59.610312012s) [noise=true , address=104.36.82.37:4663 , piece_id=XOIJUGNOKJJVWQ4226IL67H73DVYH7VBSIKI4BQSMOIGYNY7MO7A, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.023462957 (5h13m59.610519169s) [noise=true , address=104.36.82.60:4367 , piece_id=IRTAFXB5WIGLKZE4HQE7UW7J5OSZSIQK7TGY447RFZD7FCIAYQVQ, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.023666096 (5h13m59.610722308s) [noise=true , address=64.49.13.140:4511 , piece_id=A7AQ3VPKYONSOWW2YUPGWAXTMA4JPGBG4CQVYCK6PFFIFFUGPMHA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.023880075 (5h13m59.610936287s) [noise=true , address=104.36.82.23:4474 , piece_id=TDLXCBP5IPHILGHJYQD4DXYRMLL2VS2ER3ECEBAXOBTUAMR67BBA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.024106408 (5h13m59.611162629s) [noise=true , address=104.36.82.58:4269 , piece_id=4I2ROAYRNHMC53OKVWWXCS4FULL2XZ6CG5WZUPOLQ4UNELQDHHOA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.024332290 (5h13m59.611388511s) [noise=true , address=104.36.82.16:4305 , piece_id=K5YI7X35EAPFSOANSPQERKCHUOO6GDZZWKAOFNYGM6R3PA6K2LWA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.024539145 (5h13m59.611595357s) [noise=true , address=104.36.82.36:4623 , piece_id=ONIDD2KMXGVLKCD5YFBVUMMF2ORGVLFDV4LG3TRKRSINURIDAAFA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.024749439 (5h13m59.61180566s) [noise=true , address=104.36.82.43:4850 , piece_id=XWPXP7HKO2ZH22MMIODDLHM3LPIIPQP3Y3FLZ54XMMWOIN7JIADA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.024973778 (5h13m59.612029999s) [noise=true , address=45.91.116.9:4266 , piece_id=VOPNPD4GIHS2VOYBWCCTR7LX3YKL4BCZLW4XMROMAWU4WYUJ52WA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.025220337 (5h13m59.612276558s) [noise=true , address=104.36.82.15:4275 , piece_id=HCH5IC5PTQ7NCSWPJJC5OOEPEIMA34ZL6E5QPQPCDB2P2FEO6JYA, seg_pos={0 0}, upload=5382]: Uploading piece... Aug 7 21:40:46.105422677 (5h13m59.692478898s) [upload=5382]: Next returned nil segment Aug 7 21:40:46.105438417 (5h13m59.692494638s) [upload=5382]: Waiting for error group managing segments... Aug 7 21:40:46.115255601 (5h13m59.702311812s) [noise=true , address=45.91.116.9:4266 , piece_id=VOPNPD4GIHS2VOYBWCCTR7LX3YKL4BCZLW4XMROMAWU4WYUJ52WA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.123052245 (5h13m59.710108455s) [noise=true , address=104.36.82.16:4305 , piece_id=K5YI7X35EAPFSOANSPQERKCHUOO6GDZZWKAOFNYGM6R3PA6K2LWA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.123065018 (5h13m59.710121239s) [noise=true , address=104.36.82.37:4663 , piece_id=XOIJUGNOKJJVWQ4226IL67H73DVYH7VBSIKI4BQSMOIGYNY7MO7A, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.135955971 (5h13m59.723012192s) [noise=true , address=104.36.82.58:4269 , piece_id=4I2ROAYRNHMC53OKVWWXCS4FULL2XZ6CG5WZUPOLQ4UNELQDHHOA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.139757077 (5h13m59.726813288s) [noise=true , address=104.36.82.44:4891 , piece_id=D33W4EHVQ26LIOJ2KG5RKUHRZSGXAIGS2PQSIURYKGMP63RBTBGA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.148928486 (5h13m59.735984697s) [noise=true , address=104.36.82.15:4275 , piece_id=HCH5IC5PTQ7NCSWPJJC5OOEPEIMA34ZL6E5QPQPCDB2P2FEO6JYA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.149227595 (5h13m59.736283816s) [noise=true , address=104.36.82.36:4623 , piece_id=ONIDD2KMXGVLKCD5YFBVUMMF2ORGVLFDV4LG3TRKRSINURIDAAFA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.157435446 (5h13m59.744491657s) [noise=true , address=104.36.82.54:4183 , piece_id=ZOCQQFJKFATMPRR2PKHLTSWN2D3QXAMZ74K2A5KTKYQAHEGA2BUA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.193659694 (5h13m59.780715905s) [noise=true , address=104.36.82.60:4367 , piece_id=IRTAFXB5WIGLKZE4HQE7UW7J5OSZSIQK7TGY447RFZD7FCIAYQVQ, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.198544171 (5h13m59.785600382s) [noise=true , address=104.36.82.12:4172 , piece_id=2SN23SJESN46DLA25WJULJLVGE7QWLQW2QJFAUA3WUVICJ5CKLUA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.200016058 (5h13m59.787072279s) [noise=true , address=104.36.82.14:4228 , piece_id=USWUBYJMTZKFFN3DZIKQQD7PW6QAKBRUHJZRCBO4DXTHR2PQDHRQ, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.224817695 (5h13m59.811873916s) [noise=true , address=104.36.82.43:4850 , piece_id=XWPXP7HKO2ZH22MMIODDLHM3LPIIPQP3Y3FLZ54XMMWOIN7JIADA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.225279788 (5h13m59.812336009s) [noise=true , address=104.36.82.61:4383 , piece_id=ELAHQPPPJCCLCQYXDEOS62V6TOMV5NOCMUU5XZFCQ23Y6SQUWG6Q, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.305784060 (5h13m59.892840282s) [noise=true , address=104.36.82.43:4863 , piece_id=PWAZUY75NQILRAVGRSMJSFKPOPLR3WQ5O3Q47GZ5U37KXPXJ3ORQ, seg_pos={0 0}, upload=5382]: Done uploading piece. err: <nil> Aug 7 21:40:46.305806102 (5h13m59.892862323s) [seg_pos={0 0}, upload=5382]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:46.305952956 (5h13m59.893009177s) [noise=true , address=104.36.82.56:4261 , piece_id=Z27LNZSXSYSUMQXEW2DI3C54ZJK26ALV653EINMYNU6COTMAMPBA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:46.306110209 (5h13m59.89316642s) [noise=true , address=64.49.13.139:4471 , piece_id=2OSYONIOJUEZXMYFSTL7NRN7HG6KXJOHFUL7TIDW73C4VKYF3W3A, seg_pos={0 0}, upload=5382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129YkXH84aYSubinAC57swW24qCCEtvJTSMDGoxP9WdgpGcZxLr): protocol: read tcp 5.161.89.252:26750->64.49.13.139:4471: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:46.306163639 (5h13m59.89321985s) [noise=true , address=104.36.82.23:4474 , piece_id=TDLXCBP5IPHILGHJYQD4DXYRMLL2VS2ER3ECEBAXOBTUAMR67BBA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j): protocol: write tcp 5.161.89.252:22402->104.36.82.23:4474: use of closed network connection; write tcp 5.161.89.252:22402->104.36.82.23:4474: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:22402->104.36.82.23:4474: use of closed network connection Aug 7 21:40:46.306206098 (5h13m59.893262319s) [noise=true , address=64.49.13.140:4511 , piece_id=A7AQ3VPKYONSOWW2YUPGWAXTMA4JPGBG4CQVYCK6PFFIFFUGPMHA, seg_pos={0 0}, upload=5382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: read tcp 5.161.89.252:31920->64.49.13.140:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:46.306226506 (5h13m59.893282717s) [seg_pos={0 0}, upload=5382]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:46.306282762 (5h13m59.893338983s) [upload=5382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:46.306286799 (5h13m59.89334301s) [upload=5382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:57.774588980 (5h14m11.361645201s) [upload=5382]: Done uploading segments... Aug 7 21:40:57.774603057 (5h14m11.361659278s) [upload=5382]: Upload finished. err: <nil> Aug 7 21:41:10.210507562 (5h14m23.797563803s) [upload=5386]: Starting upload Aug 7 21:41:10.210589415 (5h14m23.797645655s) [upload=5386]: Done starting upload Aug 7 21:41:10.210669645 (5h14m23.797725856s) [upload=5386]: Uploading segments... Aug 7 21:41:10.210963293 (5h14m23.798019534s) [upload=5386]: Got next segment. Inline: false Aug 7 21:41:10.211025149 (5h14m23.79808139s) [upload=5386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.211035066 (5h14m23.798091308s) [upload=5386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:10.197934627 (5h15m23.784990848s) [upload=5386]: Next segment err: context deadline exceeded Aug 7 21:42:10.198070280 (5h15m23.785126501s) [upload=5386]: Done uploading segments... Aug 7 21:42:10.198139409 (5h15m23.78519562s) [upload=5386]: Upload finished. err: context deadline exceeded Duration: 1m0.001537615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [...................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001892424s FAIL (2025-08-07 21:42:07 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001914245s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X...X...X.....] duration: 114.46193ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:42:07 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 114.487927ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.000567358s FAIL (2025-08-07 21:41:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000603154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 8.048656175s FAIL (2025-08-07 21:41:49 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 5.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded Aug 7 21:39:05.437960161 (5h12m19.025016382s) [upload=5353]: Starting upload Aug 7 21:39:05.438020904 (5h12m19.025077125s) [upload=5353]: Done starting upload Aug 7 21:39:05.438137612 (5h12m19.025193863s) [upload=5353]: Uploading segments... Aug 7 21:39:05.438275971 (5h12m19.025332192s) [upload=5353]: Got next segment. Inline: false Aug 7 21:39:05.438348736 (5h12m19.025404957s) [upload=5353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:05.438353515 (5h12m19.025409736s) [upload=5353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:35.425840953 (5h12m49.012897194s) [upload=5353]: Next segment err: context deadline exceeded Aug 7 21:39:35.425899322 (5h12m49.012955573s) [upload=5353]: Done uploading segments... Aug 7 21:39:35.425916785 (5h12m49.012973036s) [upload=5353]: Upload finished. err: context deadline exceeded Aug 7 21:39:59.917805865 (5h13m13.504862107s) [upload=5368]: Starting upload Aug 7 21:39:59.917887748 (5h13m13.504943989s) [upload=5368]: Done starting upload Aug 7 21:39:59.917983418 (5h13m13.505039658s) [upload=5368]: Uploading segments... Aug 7 21:39:59.918276023 (5h13m13.505332265s) [upload=5368]: Got next segment. Inline: false Aug 7 21:39:59.918311890 (5h13m13.505368142s) [upload=5368]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:59.918320356 (5h13m13.505376598s) [upload=5368]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:29.904520509 (5h13m43.49157675s) [upload=5368]: Next segment err: context deadline exceeded Aug 7 21:40:29.904680487 (5h13m43.491736728s) [upload=5368]: Done uploading segments... Aug 7 21:40:29.904711545 (5h13m43.491767786s) [upload=5368]: Upload finished. err: context deadline exceeded Aug 7 21:40:33.289260792 (5h13m46.876317033s) [upload=5380]: Starting upload Aug 7 21:40:33.289339980 (5h13m46.876396222s) [upload=5380]: Done starting upload Aug 7 21:40:33.289423366 (5h13m46.876479606s) [upload=5380]: Uploading segments... Aug 7 21:40:33.289742541 (5h13m46.876798783s) [upload=5380]: Got next segment. Inline: false Aug 7 21:40:33.289803605 (5h13m46.876859856s) [upload=5380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:33.289812511 (5h13m46.876868763s) [upload=5380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:03.273830968 (5h14m16.860887189s) [upload=5380]: Next segment err: context deadline exceeded Aug 7 21:41:03.273965349 (5h14m16.86102156s) [upload=5380]: Done uploading segments... Aug 7 21:41:03.273981269 (5h14m16.86103749s) [upload=5380]: Upload finished. err: context deadline exceeded Duration: 8.048682403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...............................XX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.00063032s FAIL (2025-08-07 21:41:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000652139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001071229s FAIL (2025-08-07 21:41:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00108259s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.................................X........XX...X............] duration: 41.378959ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:41:45 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 41.39522ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 28.138338409s FAIL (2025-08-07 21:41:40 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 5.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded Aug 7 21:39:05.437960161 (5h12m19.025016382s) [upload=5353]: Starting upload Aug 7 21:39:05.438020904 (5h12m19.025077125s) [upload=5353]: Done starting upload Aug 7 21:39:05.438137612 (5h12m19.025193863s) [upload=5353]: Uploading segments... Aug 7 21:39:05.438275971 (5h12m19.025332192s) [upload=5353]: Got next segment. Inline: false Aug 7 21:39:05.438348736 (5h12m19.025404957s) [upload=5353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:05.438353515 (5h12m19.025409736s) [upload=5353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:35.425840953 (5h12m49.012897194s) [upload=5353]: Next segment err: context deadline exceeded Aug 7 21:39:35.425899322 (5h12m49.012955573s) [upload=5353]: Done uploading segments... Aug 7 21:39:35.425916785 (5h12m49.012973036s) [upload=5353]: Upload finished. err: context deadline exceeded Aug 7 21:39:59.917805865 (5h13m13.504862107s) [upload=5368]: Starting upload Aug 7 21:39:59.917887748 (5h13m13.504943989s) [upload=5368]: Done starting upload Aug 7 21:39:59.917983418 (5h13m13.505039658s) [upload=5368]: Uploading segments... Aug 7 21:39:59.918276023 (5h13m13.505332265s) [upload=5368]: Got next segment. Inline: false Aug 7 21:39:59.918311890 (5h13m13.505368142s) [upload=5368]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:59.918320356 (5h13m13.505376598s) [upload=5368]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:29.904520509 (5h13m43.49157675s) [upload=5368]: Next segment err: context deadline exceeded Aug 7 21:40:29.904680487 (5h13m43.491736728s) [upload=5368]: Done uploading segments... Aug 7 21:40:29.904711545 (5h13m43.491767786s) [upload=5368]: Upload finished. err: context deadline exceeded Aug 7 21:40:33.289260792 (5h13m46.876317033s) [upload=5380]: Starting upload Aug 7 21:40:33.289339980 (5h13m46.876396222s) [upload=5380]: Done starting upload Aug 7 21:40:33.289423366 (5h13m46.876479606s) [upload=5380]: Uploading segments... Aug 7 21:40:33.289742541 (5h13m46.876798783s) [upload=5380]: Got next segment. Inline: false Aug 7 21:40:33.289803605 (5h13m46.876859856s) [upload=5380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:33.289812511 (5h13m46.876868763s) [upload=5380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:03.273830968 (5h14m16.860887189s) [upload=5380]: Next segment err: context deadline exceeded Aug 7 21:41:03.273965349 (5h14m16.86102156s) [upload=5380]: Done uploading segments... Aug 7 21:41:03.273981269 (5h14m16.86103749s) [upload=5380]: Upload finished. err: context deadline exceeded Duration: 28.138358948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001198418s FAIL (2025-08-07 21:41:26 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001214899s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00079277s FAIL (2025-08-07 21:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000809171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001065126s FAIL (2025-08-07 21:41:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:15.177436931 (5h10m28.764493172s) [upload=5324]: Starting upload Aug 7 21:37:15.177533852 (5h10m28.764590103s) [upload=5324]: Done starting upload Aug 7 21:37:15.177602810 (5h10m28.764659061s) [upload=5324]: Uploading segments... Aug 7 21:37:15.177645340 (5h10m28.764701581s) [upload=5324]: Got next segment. Inline: true Aug 7 21:37:15.177650961 (5h10m28.764707202s) [upload=5324]: Waiting for error group managing segments... Aug 7 21:37:15.177698649 (5h10m28.76475489s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:15.177704300 (5h10m28.764760541s) [upload=5324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:15.177708177 (5h10m28.764764428s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:40.730185715 (5h10m54.317241957s) [upload=5324]: Done uploading segments... Aug 7 21:37:40.730203639 (5h10m54.31725988s) [upload=5324]: Upload finished. err: <nil> Aug 7 21:38:15.287715124 (5h11m28.874771365s) [upload=5341]: Starting upload Aug 7 21:38:15.287780276 (5h11m28.874836516s) [upload=5341]: Done starting upload Aug 7 21:38:15.287844856 (5h11m28.874901107s) [upload=5341]: Uploading segments... Aug 7 21:38:15.287889961 (5h11m28.874946201s) [upload=5341]: Got next segment. Inline: true Aug 7 21:38:15.287895600 (5h11m28.874951842s) [upload=5341]: Waiting for error group managing segments... Aug 7 21:38:15.287984066 (5h11m28.875040317s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:15.287990077 (5h11m28.875046318s) [upload=5341]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:15.287993564 (5h11m28.875049804s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:15.275679218 (5h12m28.862735459s) [upload=5341]: Done uploading segments... Aug 7 21:39:15.275708622 (5h12m28.862764883s) [upload=5341]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:15.358605080 (5h12m28.945661322s) [upload=5358]: Starting upload Aug 7 21:39:15.358690641 (5h12m28.945746882s) [upload=5358]: Done starting upload Aug 7 21:39:15.358748869 (5h12m28.945805111s) [upload=5358]: Uploading segments... Aug 7 21:39:15.358790938 (5h12m28.94584718s) [upload=5358]: Got next segment. Inline: true Aug 7 21:39:15.358796779 (5h12m28.94585302s) [upload=5358]: Waiting for error group managing segments... Aug 7 21:39:15.358840480 (5h12m28.945896721s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:15.358846441 (5h12m28.945902683s) [upload=5358]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:15.358849918 (5h12m28.945906159s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:15.344303433 (5h13m28.931359654s) [upload=5358]: Done uploading segments... Aug 7 21:40:15.344323120 (5h13m28.931379331s) [upload=5358]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:15.438334176 (5h13m29.025390397s) [upload=5373]: Starting upload Aug 7 21:40:15.438379379 (5h13m29.025435601s) [upload=5373]: Done starting upload Aug 7 21:40:15.438418503 (5h13m29.025474714s) [upload=5373]: Uploading segments... Aug 7 21:40:15.438447206 (5h13m29.025503427s) [upload=5373]: Got next segment. Inline: true Aug 7 21:40:15.438450683 (5h13m29.025506894s) [upload=5373]: Waiting for error group managing segments... Aug 7 21:40:15.438481920 (5h13m29.025538141s) [upload=5373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:15.438487511 (5h13m29.025543722s) [upload=5373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:15.438489826 (5h13m29.025546047s) [upload=5373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:15.423387101 (5h14m29.010443342s) [upload=5373]: Done uploading segments... Aug 7 21:41:15.423408011 (5h14m29.010464251s) [upload=5373]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001079423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 771.985645ms FAIL (2025-08-07 21:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 772.011683ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXX.] duration: 33.910974787s FAIL (2025-08-07 21:41:10 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45575 (5h12m48.424901796s) [upload=5363]: Got next segment. Inline: false Aug 7 21:39:34.837870520 (5h12m48.424926741s) [upload=5363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:34.837874097 (5h12m48.424930318s) [upload=5363]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:49.878304965 (5h13m3.465361177s) [seg_pos={0 0}, upload=5363]: Begin upload segment... Aug 7 21:39:49.878344229 (5h13m3.46540045s) [seg_pos={0 0}, upload=5363]: Done begin upload segment. Aug 7 21:39:49.878368674 (5h13m3.465424885s) [noise=true , address=64.49.13.138:4485 , piece_id=H2XJZ3TWBUP6CUVYN54VYKBOOU3HVWO2EHHDEKZVS4NEC7MHUSLQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878377310 (5h13m3.465433531s) [noise=true , address=79.127.219.51:4181 , piece_id=AGQNJQ766BO4W2TX7NBJQMM6CAKH7IFVX7TUNVNHDH3Q6N7CLBLQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878659107 (5h13m3.465715328s) [noise=true , address=104.36.82.44:4885 , piece_id=56VL4UKYMTBIRCYWTWHRJTY7DSD3OIESWBBABB5AML62BY4GZTVA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878665629 (5h13m3.46572185s) [noise=true , address=104.36.82.38:4701 , piece_id=MJ3UQAIQROLNLNUCADQ7QTOL3TX7QCWTXMMO2M6NTIIYWIWCI7NA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878876833 (5h13m3.465933054s) [noise=true , address=79.127.220.97:4396 , piece_id=3ERXXKPQO2TBIFDGQCPTXS44BX46O437IK6ACLATSPRZ5NTSQDMQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878941813 (5h13m3.465998035s) [noise=true , address=104.36.82.60:4370 , piece_id=U5D4WXXW5QIS2IWOJMAZMNY6XYTW3PMK2V3FYXA7IOXXPHHIO7EQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879085081 (5h13m3.466141302s) [noise=true , address=104.36.82.59:4335 , piece_id=IF7CL65N352T52WGOOO57RGVRK2GDOFC35SQ2YZS7QVJE3MV7PGA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879196810 (5h13m3.466253021s) [noise=true , address=104.36.82.62:4436 , piece_id=Y2K4RLVGID33T6CCN4RJBMELZGGDQTL6COXVGHRGRUHWC34YPMGQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879303479 (5h13m3.4663597s) [noise=true , address=104.36.82.15:4280 , piece_id=KP7GTIDO3HE26XQPQQTUQ6ZSYWRNG364U5UR6Y7GNV3WTXXGDSIQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879430125 (5h13m3.466486346s) [noise=true , address=79.127.205.233:4638 , piece_id=DOK5RP6ECKFO427CS5KO2T225DKU5SUNTULES7YUPS2T7BCY2WHQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879511177 (5h13m3.466567417s) [noise=true , address=104.36.82.41:4765 , piece_id=T77DVJNUDYDARPAGPEDLGA2RK5Y3XAC32BXBKS3Z34XJECE46BZA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879689129 (5h13m3.466745371s) [noise=true , address=45.91.116.7:4032 , piece_id=6I25VWXSRXZXOZYGSBONI3DQ6YI3WRVWYB6JNRKN56IJ4W7DBZLA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879718113 (5h13m3.466774323s) [noise=true , address=45.91.116.9:4266 , piece_id=NOZPHSZTEXCYT6W6SZWQWF52COQK6VZRY3KE2W5QFYRYFXTVVP7A, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880043561 (5h13m3.467099792s) [noise=true , address=104.36.82.39:4743 , piece_id=OXYPTNFAFEUPYYWRD6XK4NFGOIAGBAR7ANTUXPKK7EFIKYPXXQWQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880107590 (5h13m3.467163811s) [noise=true , address=104.36.82.18:4382 , piece_id=NKWDSUETR7DYMLP7WT4CEL3BQZVHAMGXZJXMDUBTRPER3AAAQWKA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880341958 (5h13m3.467398189s) [noise=true , address=64.49.13.154:4926 , piece_id=4CYHDAPIDZZRXPNZWO5C4UOJYSXUKI34EQNSCRFQTLJUSZB2PEKA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880457353 (5h13m3.467513584s) [noise=true , address=104.36.82.34:4557 , piece_id=CAX3GHRMDEXQVQZE5T6MNNRIEF2WBFH2Z4MRAPYQIOFRQQ5YPB2Q, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.881788197 (5h13m3.468844418s) [noise=true , address=104.36.82.52:4086 , piece_id=DVBDLADMVQNIJOWCBHHQLQFLOCJWNC7BJQKXNFIO5IDCEZGVJPOA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.972772343 (5h13m3.559828564s) [upload=5363]: Next returned nil segment Aug 7 21:39:49.972806465 (5h13m3.559862686s) [upload=5363]: Waiting for error group managing segments... Aug 7 21:39:49.980432491 (5h13m3.567488712s) [noise=true , address=79.127.219.51:4181 , piece_id=AGQNJQ766BO4W2TX7NBJQMM6CAKH7IFVX7TUNVNHDH3Q6N7CLBLQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.984992463 (5h13m3.572048684s) [noise=true , address=104.36.82.34:4557 , piece_id=CAX3GHRMDEXQVQZE5T6MNNRIEF2WBFH2Z4MRAPYQIOFRQQ5YPB2Q, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.986167406 (5h13m3.573223617s) [noise=true , address=104.36.82.38:4701 , piece_id=MJ3UQAIQROLNLNUCADQ7QTOL3TX7QCWTXMMO2M6NTIIYWIWCI7NA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.995474859 (5h13m3.582531071s) [noise=true , address=45.91.116.7:4032 , piece_id=6I25VWXSRXZXOZYGSBONI3DQ6YI3WRVWYB6JNRKN56IJ4W7DBZLA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.995633937 (5h13m3.582690158s) [noise=true , address=45.91.116.9:4266 , piece_id=NOZPHSZTEXCYT6W6SZWQWF52COQK6VZRY3KE2W5QFYRYFXTVVP7A, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.002825401 (5h13m3.589881622s) [noise=true , address=104.36.82.18:4382 , piece_id=NKWDSUETR7DYMLP7WT4CEL3BQZVHAMGXZJXMDUBTRPER3AAAQWKA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.008951135 (5h13m3.596007356s) [noise=true , address=104.36.82.62:4436 , piece_id=Y2K4RLVGID33T6CCN4RJBMELZGGDQTL6COXVGHRGRUHWC34YPMGQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.009865722 (5h13m3.596921933s) [noise=true , address=104.36.82.41:4765 , piece_id=T77DVJNUDYDARPAGPEDLGA2RK5Y3XAC32BXBKS3Z34XJECE46BZA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.011287307 (5h13m3.598343528s) [noise=true , address=104.36.82.15:4280 , piece_id=KP7GTIDO3HE26XQPQQTUQ6ZSYWRNG364U5UR6Y7GNV3WTXXGDSIQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.013773407 (5h13m3.600829618s) [noise=true , address=104.36.82.60:4370 , piece_id=U5D4WXXW5QIS2IWOJMAZMNY6XYTW3PMK2V3FYXA7IOXXPHHIO7EQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.024686839 (5h13m3.61174306s) [noise=true , address=104.36.82.39:4743 , piece_id=OXYPTNFAFEUPYYWRD6XK4NFGOIAGBAR7ANTUXPKK7EFIKYPXXQWQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.040897448 (5h13m3.627953669s) [noise=true , address=104.36.82.52:4086 , piece_id=DVBDLADMVQNIJOWCBHHQLQFLOCJWNC7BJQKXNFIO5IDCEZGVJPOA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.066965858 (5h13m3.654022069s) [noise=true , address=79.127.220.97:4396 , piece_id=3ERXXKPQO2TBIFDGQCPTXS44BX46O437IK6ACLATSPRZ5NTSQDMQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.107738957 (5h13m3.694795198s) [noise=true , address=104.36.82.59:4335 , piece_id=IF7CL65N352T52WGOOO57RGVRK2GDOFC35SQ2YZS7QVJE3MV7PGA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.107779223 (5h13m3.694835463s) [seg_pos={0 0}, upload=5363]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:50.108205358 (5h13m3.695261579s) [noise=true , address=64.49.13.138:4485 , piece_id=H2XJZ3TWBUP6CUVYN54VYKBOOU3HVWO2EHHDEKZVS4NEC7MHUSLQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jw25xoDRqePr6VBUBLYmMvzN52nM5E6zgf4xLDztWuPRwZAUM): protocol: read tcp 5.161.89.252:45602->64.49.13.138:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108409869 (5h13m3.69546609s) [noise=true , address=104.36.82.44:4885 , piece_id=56VL4UKYMTBIRCYWTWHRJTY7DSD3OIESWBBABB5AML62BY4GZTVA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: read tcp 5.161.89.252:28322->104.36.82.44:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108477034 (5h13m3.695533275s) [noise=true , address=79.127.205.233:4638 , piece_id=DOK5RP6ECKFO427CS5KO2T225DKU5SUNTULES7YUPS2T7BCY2WHQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DcsEnrgLd6q2HtcfWGEjLb5qUyt3iUpTxTYW2oeVnBD43ySHc): protocol: read tcp 5.161.89.252:58494->79.127.205.233:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108501380 (5h13m3.695557601s) [noise=true , address=64.49.13.154:4926 , piece_id=4CYHDAPIDZZRXPNZWO5C4UOJYSXUKI34EQNSCRFQTLJUSZB2PEKA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bpEYLHxV7TM3KdheTwmwBxpPvm3PKccMVFLfLc5fTXtJZX5VY): protocol: read tcp 5.161.89.252:54010->64.49.13.154:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108539411 (5h13m3.695595632s) [seg_pos={0 0}, upload=5363]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:50.108677649 (5h13m3.69573386s) [upload=5363]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:50.108684242 (5h13m3.695740453s) [upload=5363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:36.145495594 (5h13m49.732551845s) [upload=5363]: Done uploading segments... Aug 7 21:40:36.145521862 (5h13m49.732578103s) [upload=5363]: Upload finished. err: metaclient: context deadline exceeded Duration: 33.910993048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 19.04583511s FAIL (2025-08-07 21:41:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.04585654s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000564994s FAIL (2025-08-07 21:41:06 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== hed. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:26.513648977 (5h11m40.100705218s) [upload=5344]: Starting upload Aug 7 21:38:26.513731310 (5h11m40.100787552s) [upload=5344]: Done starting upload Aug 7 21:38:26.513792245 (5h11m40.100848496s) [upload=5344]: Uploading segments... Aug 7 21:38:26.513834112 (5h11m40.100890354s) [upload=5344]: Got next segment. Inline: true Aug 7 21:38:26.513839653 (5h11m40.100895904s) [upload=5344]: Waiting for error group managing segments... Aug 7 21:38:26.513884006 (5h11m40.100940247s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:26.513889947 (5h11m40.100946188s) [upload=5344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:26.513893434 (5h11m40.100949675s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:56.502052245 (5h12m10.089108537s) [upload=5344]: Done uploading segments... Aug 7 21:38:56.502102730 (5h12m10.089158972s) [upload=5344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:24.008006172 (5h12m37.595062394s) [upload=5359]: Starting upload Aug 7 21:39:24.008059211 (5h12m37.595115422s) [upload=5359]: Done starting upload Aug 7 21:39:24.008103604 (5h12m37.595159835s) [upload=5359]: Uploading segments... Aug 7 21:39:24.008156191 (5h12m37.595212413s) [upload=5359]: Got next segment. Inline: true Aug 7 21:39:24.008162193 (5h12m37.595218414s) [upload=5359]: Waiting for error group managing segments... Aug 7 21:39:24.008207327 (5h12m37.595263548s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:24.008212668 (5h12m37.595268878s) [upload=5359]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:24.008216064 (5h12m37.595272285s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:53.997226503 (5h13m7.584282744s) [upload=5359]: Done uploading segments... Aug 7 21:39:53.997270334 (5h13m7.584326586s) [upload=5359]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:36.779619947 (5h13m50.366676188s) [upload=5383]: Starting upload Aug 7 21:40:36.779696139 (5h13m50.36675238s) [upload=5383]: Done starting upload Aug 7 21:40:36.779752775 (5h13m50.366809016s) [upload=5383]: Uploading segments... Aug 7 21:40:36.779788803 (5h13m50.366845044s) [upload=5383]: Got next segment. Inline: true Aug 7 21:40:36.779794423 (5h13m50.366850655s) [upload=5383]: Waiting for error group managing segments... Aug 7 21:40:36.779835950 (5h13m50.366892191s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:36.779841010 (5h13m50.366897241s) [upload=5383]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:36.779844195 (5h13m50.366900437s) [upload=5383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:06.765292852 (5h14m20.352349064s) [upload=5383]: Done uploading segments... Aug 7 21:41:06.765314052 (5h14m20.352370273s) [upload=5383]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000586774s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X...X...X....] duration: 109.535954ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:41:06 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 109.549188ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000561917s FAIL (2025-08-07 21:41:03 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 5.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded Aug 7 21:39:05.437960161 (5h12m19.025016382s) [upload=5353]: Starting upload Aug 7 21:39:05.438020904 (5h12m19.025077125s) [upload=5353]: Done starting upload Aug 7 21:39:05.438137612 (5h12m19.025193863s) [upload=5353]: Uploading segments... Aug 7 21:39:05.438275971 (5h12m19.025332192s) [upload=5353]: Got next segment. Inline: false Aug 7 21:39:05.438348736 (5h12m19.025404957s) [upload=5353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:05.438353515 (5h12m19.025409736s) [upload=5353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:35.425840953 (5h12m49.012897194s) [upload=5353]: Next segment err: context deadline exceeded Aug 7 21:39:35.425899322 (5h12m49.012955573s) [upload=5353]: Done uploading segments... Aug 7 21:39:35.425916785 (5h12m49.012973036s) [upload=5353]: Upload finished. err: context deadline exceeded Aug 7 21:39:59.917805865 (5h13m13.504862107s) [upload=5368]: Starting upload Aug 7 21:39:59.917887748 (5h13m13.504943989s) [upload=5368]: Done starting upload Aug 7 21:39:59.917983418 (5h13m13.505039658s) [upload=5368]: Uploading segments... Aug 7 21:39:59.918276023 (5h13m13.505332265s) [upload=5368]: Got next segment. Inline: false Aug 7 21:39:59.918311890 (5h13m13.505368142s) [upload=5368]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:59.918320356 (5h13m13.505376598s) [upload=5368]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:29.904520509 (5h13m43.49157675s) [upload=5368]: Next segment err: context deadline exceeded Aug 7 21:40:29.904680487 (5h13m43.491736728s) [upload=5368]: Done uploading segments... Aug 7 21:40:29.904711545 (5h13m43.491767786s) [upload=5368]: Upload finished. err: context deadline exceeded Aug 7 21:40:33.289260792 (5h13m46.876317033s) [upload=5380]: Starting upload Aug 7 21:40:33.289339980 (5h13m46.876396222s) [upload=5380]: Done starting upload Aug 7 21:40:33.289423366 (5h13m46.876479606s) [upload=5380]: Uploading segments... Aug 7 21:40:33.289742541 (5h13m46.876798783s) [upload=5380]: Got next segment. Inline: false Aug 7 21:40:33.289803605 (5h13m46.876859856s) [upload=5380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:33.289812511 (5h13m46.876868763s) [upload=5380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:03.273830968 (5h14m16.860887189s) [upload=5380]: Next segment err: context deadline exceeded Aug 7 21:41:03.273965349 (5h14m16.86102156s) [upload=5380]: Done uploading segments... Aug 7 21:41:03.273981269 (5h14m16.86103749s) [upload=5380]: Upload finished. err: context deadline exceeded Duration: 30.000581525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m49.014258147s FAIL (2025-08-07 21:40:56 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m49.014273876s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [....................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000810821s FAIL (2025-08-07 21:40:52 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000827362s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [................................XX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000698876s FAIL (2025-08-07 21:40:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000713564s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [..................................X........XX...X...........] duration: 26.456415ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:40:38 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 26.477774ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m1.320827058s FAIL (2025-08-07 21:40:36 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45575 (5h12m48.424901796s) [upload=5363]: Got next segment. Inline: false Aug 7 21:39:34.837870520 (5h12m48.424926741s) [upload=5363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:34.837874097 (5h12m48.424930318s) [upload=5363]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:49.878304965 (5h13m3.465361177s) [seg_pos={0 0}, upload=5363]: Begin upload segment... Aug 7 21:39:49.878344229 (5h13m3.46540045s) [seg_pos={0 0}, upload=5363]: Done begin upload segment. Aug 7 21:39:49.878368674 (5h13m3.465424885s) [noise=true , address=64.49.13.138:4485 , piece_id=H2XJZ3TWBUP6CUVYN54VYKBOOU3HVWO2EHHDEKZVS4NEC7MHUSLQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878377310 (5h13m3.465433531s) [noise=true , address=79.127.219.51:4181 , piece_id=AGQNJQ766BO4W2TX7NBJQMM6CAKH7IFVX7TUNVNHDH3Q6N7CLBLQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878659107 (5h13m3.465715328s) [noise=true , address=104.36.82.44:4885 , piece_id=56VL4UKYMTBIRCYWTWHRJTY7DSD3OIESWBBABB5AML62BY4GZTVA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878665629 (5h13m3.46572185s) [noise=true , address=104.36.82.38:4701 , piece_id=MJ3UQAIQROLNLNUCADQ7QTOL3TX7QCWTXMMO2M6NTIIYWIWCI7NA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878876833 (5h13m3.465933054s) [noise=true , address=79.127.220.97:4396 , piece_id=3ERXXKPQO2TBIFDGQCPTXS44BX46O437IK6ACLATSPRZ5NTSQDMQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.878941813 (5h13m3.465998035s) [noise=true , address=104.36.82.60:4370 , piece_id=U5D4WXXW5QIS2IWOJMAZMNY6XYTW3PMK2V3FYXA7IOXXPHHIO7EQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879085081 (5h13m3.466141302s) [noise=true , address=104.36.82.59:4335 , piece_id=IF7CL65N352T52WGOOO57RGVRK2GDOFC35SQ2YZS7QVJE3MV7PGA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879196810 (5h13m3.466253021s) [noise=true , address=104.36.82.62:4436 , piece_id=Y2K4RLVGID33T6CCN4RJBMELZGGDQTL6COXVGHRGRUHWC34YPMGQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879303479 (5h13m3.4663597s) [noise=true , address=104.36.82.15:4280 , piece_id=KP7GTIDO3HE26XQPQQTUQ6ZSYWRNG364U5UR6Y7GNV3WTXXGDSIQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879430125 (5h13m3.466486346s) [noise=true , address=79.127.205.233:4638 , piece_id=DOK5RP6ECKFO427CS5KO2T225DKU5SUNTULES7YUPS2T7BCY2WHQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879511177 (5h13m3.466567417s) [noise=true , address=104.36.82.41:4765 , piece_id=T77DVJNUDYDARPAGPEDLGA2RK5Y3XAC32BXBKS3Z34XJECE46BZA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879689129 (5h13m3.466745371s) [noise=true , address=45.91.116.7:4032 , piece_id=6I25VWXSRXZXOZYGSBONI3DQ6YI3WRVWYB6JNRKN56IJ4W7DBZLA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.879718113 (5h13m3.466774323s) [noise=true , address=45.91.116.9:4266 , piece_id=NOZPHSZTEXCYT6W6SZWQWF52COQK6VZRY3KE2W5QFYRYFXTVVP7A, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880043561 (5h13m3.467099792s) [noise=true , address=104.36.82.39:4743 , piece_id=OXYPTNFAFEUPYYWRD6XK4NFGOIAGBAR7ANTUXPKK7EFIKYPXXQWQ, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880107590 (5h13m3.467163811s) [noise=true , address=104.36.82.18:4382 , piece_id=NKWDSUETR7DYMLP7WT4CEL3BQZVHAMGXZJXMDUBTRPER3AAAQWKA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880341958 (5h13m3.467398189s) [noise=true , address=64.49.13.154:4926 , piece_id=4CYHDAPIDZZRXPNZWO5C4UOJYSXUKI34EQNSCRFQTLJUSZB2PEKA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.880457353 (5h13m3.467513584s) [noise=true , address=104.36.82.34:4557 , piece_id=CAX3GHRMDEXQVQZE5T6MNNRIEF2WBFH2Z4MRAPYQIOFRQQ5YPB2Q, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.881788197 (5h13m3.468844418s) [noise=true , address=104.36.82.52:4086 , piece_id=DVBDLADMVQNIJOWCBHHQLQFLOCJWNC7BJQKXNFIO5IDCEZGVJPOA, seg_pos={0 0}, upload=5363]: Uploading piece... Aug 7 21:39:49.972772343 (5h13m3.559828564s) [upload=5363]: Next returned nil segment Aug 7 21:39:49.972806465 (5h13m3.559862686s) [upload=5363]: Waiting for error group managing segments... Aug 7 21:39:49.980432491 (5h13m3.567488712s) [noise=true , address=79.127.219.51:4181 , piece_id=AGQNJQ766BO4W2TX7NBJQMM6CAKH7IFVX7TUNVNHDH3Q6N7CLBLQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.984992463 (5h13m3.572048684s) [noise=true , address=104.36.82.34:4557 , piece_id=CAX3GHRMDEXQVQZE5T6MNNRIEF2WBFH2Z4MRAPYQIOFRQQ5YPB2Q, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.986167406 (5h13m3.573223617s) [noise=true , address=104.36.82.38:4701 , piece_id=MJ3UQAIQROLNLNUCADQ7QTOL3TX7QCWTXMMO2M6NTIIYWIWCI7NA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.995474859 (5h13m3.582531071s) [noise=true , address=45.91.116.7:4032 , piece_id=6I25VWXSRXZXOZYGSBONI3DQ6YI3WRVWYB6JNRKN56IJ4W7DBZLA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:49.995633937 (5h13m3.582690158s) [noise=true , address=45.91.116.9:4266 , piece_id=NOZPHSZTEXCYT6W6SZWQWF52COQK6VZRY3KE2W5QFYRYFXTVVP7A, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.002825401 (5h13m3.589881622s) [noise=true , address=104.36.82.18:4382 , piece_id=NKWDSUETR7DYMLP7WT4CEL3BQZVHAMGXZJXMDUBTRPER3AAAQWKA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.008951135 (5h13m3.596007356s) [noise=true , address=104.36.82.62:4436 , piece_id=Y2K4RLVGID33T6CCN4RJBMELZGGDQTL6COXVGHRGRUHWC34YPMGQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.009865722 (5h13m3.596921933s) [noise=true , address=104.36.82.41:4765 , piece_id=T77DVJNUDYDARPAGPEDLGA2RK5Y3XAC32BXBKS3Z34XJECE46BZA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.011287307 (5h13m3.598343528s) [noise=true , address=104.36.82.15:4280 , piece_id=KP7GTIDO3HE26XQPQQTUQ6ZSYWRNG364U5UR6Y7GNV3WTXXGDSIQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.013773407 (5h13m3.600829618s) [noise=true , address=104.36.82.60:4370 , piece_id=U5D4WXXW5QIS2IWOJMAZMNY6XYTW3PMK2V3FYXA7IOXXPHHIO7EQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.024686839 (5h13m3.61174306s) [noise=true , address=104.36.82.39:4743 , piece_id=OXYPTNFAFEUPYYWRD6XK4NFGOIAGBAR7ANTUXPKK7EFIKYPXXQWQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.040897448 (5h13m3.627953669s) [noise=true , address=104.36.82.52:4086 , piece_id=DVBDLADMVQNIJOWCBHHQLQFLOCJWNC7BJQKXNFIO5IDCEZGVJPOA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.066965858 (5h13m3.654022069s) [noise=true , address=79.127.220.97:4396 , piece_id=3ERXXKPQO2TBIFDGQCPTXS44BX46O437IK6ACLATSPRZ5NTSQDMQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.107738957 (5h13m3.694795198s) [noise=true , address=104.36.82.59:4335 , piece_id=IF7CL65N352T52WGOOO57RGVRK2GDOFC35SQ2YZS7QVJE3MV7PGA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: <nil> Aug 7 21:39:50.107779223 (5h13m3.694835463s) [seg_pos={0 0}, upload=5363]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:50.108205358 (5h13m3.695261579s) [noise=true , address=64.49.13.138:4485 , piece_id=H2XJZ3TWBUP6CUVYN54VYKBOOU3HVWO2EHHDEKZVS4NEC7MHUSLQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jw25xoDRqePr6VBUBLYmMvzN52nM5E6zgf4xLDztWuPRwZAUM): protocol: read tcp 5.161.89.252:45602->64.49.13.138:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108409869 (5h13m3.69546609s) [noise=true , address=104.36.82.44:4885 , piece_id=56VL4UKYMTBIRCYWTWHRJTY7DSD3OIESWBBABB5AML62BY4GZTVA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: read tcp 5.161.89.252:28322->104.36.82.44:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108477034 (5h13m3.695533275s) [noise=true , address=79.127.205.233:4638 , piece_id=DOK5RP6ECKFO427CS5KO2T225DKU5SUNTULES7YUPS2T7BCY2WHQ, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DcsEnrgLd6q2HtcfWGEjLb5qUyt3iUpTxTYW2oeVnBD43ySHc): protocol: read tcp 5.161.89.252:58494->79.127.205.233:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108501380 (5h13m3.695557601s) [noise=true , address=64.49.13.154:4926 , piece_id=4CYHDAPIDZZRXPNZWO5C4UOJYSXUKI34EQNSCRFQTLJUSZB2PEKA, seg_pos={0 0}, upload=5363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bpEYLHxV7TM3KdheTwmwBxpPvm3PKccMVFLfLc5fTXtJZX5VY): protocol: read tcp 5.161.89.252:54010->64.49.13.154:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:50.108539411 (5h13m3.695595632s) [seg_pos={0 0}, upload=5363]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:50.108677649 (5h13m3.69573386s) [upload=5363]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:50.108684242 (5h13m3.695740453s) [upload=5363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:36.145495594 (5h13m49.732551845s) [upload=5363]: Done uploading segments... Aug 7 21:40:36.145521862 (5h13m49.732578103s) [upload=5363]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m1.320841003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000981293s FAIL (2025-08-07 21:40:29 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 21:31:11.924580593 (5h4m25.511636844s) [upload=5227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.924591073 (5h4m25.511647324s) [upload=5227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:41.901407105 (5h4m55.488463316s) [upload=5227]: Next segment err: context deadline exceeded Aug 7 21:31:41.901545804 (5h4m55.488602015s) [upload=5227]: Done uploading segments... Aug 7 21:31:41.901568547 (5h4m55.488624747s) [upload=5227]: Upload finished. err: context deadline exceeded Aug 7 21:32:25.036046257 (5h5m38.623102498s) [upload=5243]: Starting upload Aug 7 21:32:25.036157154 (5h5m38.623213395s) [upload=5243]: Done starting upload Aug 7 21:32:25.036227865 (5h5m38.623284077s) [upload=5243]: Uploading segments... Aug 7 21:32:25.036538426 (5h5m38.623594667s) [upload=5243]: Got next segment. Inline: false Aug 7 21:32:25.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded Aug 7 21:39:05.437960161 (5h12m19.025016382s) [upload=5353]: Starting upload Aug 7 21:39:05.438020904 (5h12m19.025077125s) [upload=5353]: Done starting upload Aug 7 21:39:05.438137612 (5h12m19.025193863s) [upload=5353]: Uploading segments... Aug 7 21:39:05.438275971 (5h12m19.025332192s) [upload=5353]: Got next segment. Inline: false Aug 7 21:39:05.438348736 (5h12m19.025404957s) [upload=5353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:05.438353515 (5h12m19.025409736s) [upload=5353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:35.425840953 (5h12m49.012897194s) [upload=5353]: Next segment err: context deadline exceeded Aug 7 21:39:35.425899322 (5h12m49.012955573s) [upload=5353]: Done uploading segments... Aug 7 21:39:35.425916785 (5h12m49.012973036s) [upload=5353]: Upload finished. err: context deadline exceeded Aug 7 21:39:59.917805865 (5h13m13.504862107s) [upload=5368]: Starting upload Aug 7 21:39:59.917887748 (5h13m13.504943989s) [upload=5368]: Done starting upload Aug 7 21:39:59.917983418 (5h13m13.505039658s) [upload=5368]: Uploading segments... Aug 7 21:39:59.918276023 (5h13m13.505332265s) [upload=5368]: Got next segment. Inline: false Aug 7 21:39:59.918311890 (5h13m13.505368142s) [upload=5368]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:59.918320356 (5h13m13.505376598s) [upload=5368]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:29.904520509 (5h13m43.49157675s) [upload=5368]: Next segment err: context deadline exceeded Aug 7 21:40:29.904680487 (5h13m43.491736728s) [upload=5368]: Done uploading segments... Aug 7 21:40:29.904711545 (5h13m43.491767786s) [upload=5368]: Upload finished. err: context deadline exceeded Duration: 30.001002193s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001254071s FAIL (2025-08-07 21:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00139319s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.796978034s FAIL (2025-08-07 21:40:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 2.797001408s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.00136197s FAIL (2025-08-07 21:40:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5h3m18.749276703s) [upload=5191]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:05.259129610 (5h3m18.846185831s) [upload=5205]: Starting upload Aug 7 21:30:05.259178532 (5h3m18.846234742s) [upload=5205]: Done starting upload Aug 7 21:30:05.259225909 (5h3m18.846282131s) [upload=5205]: Uploading segments... Aug 7 21:30:05.259254723 (5h3m18.846310935s) [upload=5205]: Got next segment. Inline: true Aug 7 21:30:05.259258370 (5h3m18.846314591s) [upload=5205]: Waiting for error group managing segments... Aug 7 21:30:05.259289909 (5h3m18.846346131s) [upload=5205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:05.259293515 (5h3m18.846349736s) [upload=5205]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:05.259295659 (5h3m18.84635187s) [upload=5205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:05.245621000 (5h4m18.83267725s) [upload=5205]: Done uploading segments... Aug 7 21:31:05.245643692 (5h4m18.832699933s) [upload=5205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:05.321541416 (5h4m18.908597658s) [upload=5224]: Starting upload Aug 7 21:31:05.321629561 (5h4m18.908685802s) [upload=5224]: Done starting upload Aug 7 21:31:05.321697208 (5h4m18.908753448s) [upload=5224]: Uploading segments... Aug 7 21:31:05.321740748 (5h4m18.908796989s) [upload=5224]: Got next segment. Inline: true Aug 7 21:31:05.321746479 (5h4m18.90880272s) [upload=5224]: Waiting for error group managing segments... Aug 7 21:31:05.321792836 (5h4m18.908849077s) [upload=5224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:05.321798496 (5h4m18.908854738s) [upload=5224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:05.321802404 (5h4m18.908858645s) [upload=5224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:05.310572656 (5h5m18.897628866s) [upload=5224]: Done uploading segments... Aug 7 21:32:05.310588175 (5h5m18.897644386s) [upload=5224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:05.391427942 (5h5m18.978484152s) [upload=5237]: Starting upload Aug 7 21:32:05.391496580 (5h5m18.978552801s) [upload=5237]: Done starting upload Aug 7 21:32:05.391539640 (5h5m18.978595861s) [upload=5237]: Uploading segments... Aug 7 21:32:05.391566651 (5h5m18.978622872s) [upload=5237]: Got next segment. Inline: true Aug 7 21:32:05.391570338 (5h5m18.978626559s) [upload=5237]: Waiting for error group managing segments... Aug 7 21:32:05.391600825 (5h5m18.978657046s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:05.391604532 (5h5m18.978660753s) [upload=5237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:05.391607247 (5h5m18.978663468s) [upload=5237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:05.380269232 (5h6m18.967325453s) [upload=5237]: Done uploading segments... Aug 7 21:33:05.380285241 (5h6m18.967341462s) [upload=5237]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:09.996864149 (5h6m23.58392037s) [upload=5253]: Starting upload Aug 7 21:33:09.996931725 (5h6m23.583987946s) [upload=5253]: Done starting upload Aug 7 21:33:09.996981849 (5h6m23.58403807s) [upload=5253]: Uploading segments... Aug 7 21:33:09.997031481 (5h6m23.584087702s) [upload=5253]: Got next segment. Inline: true Aug 7 21:33:09.997035519 (5h6m23.58409174s) [upload=5253]: Waiting for error group managing segments... Aug 7 21:33:09.997069202 (5h6m23.584125423s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:09.997073270 (5h6m23.58412948s) [upload=5253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:09.997075754 (5h6m23.584131965s) [upload=5253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:09.982031009 (5h7m23.56908725s) [upload=5253]: Done uploading segments... Aug 7 21:34:09.982059972 (5h7m23.569116213s) [upload=5253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:10.069393604 (5h7m23.656449814s) [upload=5272]: Starting upload Aug 7 21:34:10.069444327 (5h7m23.656500539s) [upload=5272]: Done starting upload Aug 7 21:34:10.069485265 (5h7m23.656541486s) [upload=5272]: Uploading segments... Aug 7 21:34:10.069514979 (5h7m23.65657119s) [upload=5272]: Got next segment. Inline: true Aug 7 21:34:10.069518766 (5h7m23.656574978s) [upload=5272]: Waiting for error group managing segments... Aug 7 21:34:10.069549243 (5h7m23.656605464s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:10.069552840 (5h7m23.656609061s) [upload=5272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:10.069555245 (5h7m23.656611456s) [upload=5272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:10.049493169 (5h8m23.63654939s) [upload=5272]: Done uploading segments... Aug 7 21:35:10.049531101 (5h8m23.636587322s) [upload=5272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:15.032246862 (5h8m28.619303102s) [upload=5292]: Starting upload Aug 7 21:35:15.032327261 (5h8m28.619383502s) [upload=5292]: Done starting upload Aug 7 21:35:15.032391071 (5h8m28.619447312s) [upload=5292]: Uploading segments... Aug 7 21:35:15.032434000 (5h8m28.619490242s) [upload=5292]: Got next segment. Inline: true Aug 7 21:35:15.032439461 (5h8m28.619495711s) [upload=5292]: Waiting for error group managing segments... Aug 7 21:35:15.032484926 (5h8m28.619541167s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:15.032491267 (5h8m28.619547509s) [upload=5292]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:15.032494834 (5h8m28.619551085s) [upload=5292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:15.017690007 (5h9m28.604746248s) [upload=5292]: Done uploading segments... Aug 7 21:36:15.017709373 (5h9m28.604765615s) [upload=5292]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:15.108542356 (5h9m28.695598567s) [upload=5309]: Starting upload Aug 7 21:36:15.108597850 (5h9m28.695654061s) [upload=5309]: Done starting upload Aug 7 21:36:15.108645098 (5h9m28.695701319s) [upload=5309]: Uploading segments... Aug 7 21:36:15.108673942 (5h9m28.695730153s) [upload=5309]: Got next segment. Inline: true Aug 7 21:36:15.108677278 (5h9m28.695733499s) [upload=5309]: Waiting for error group managing segments... Aug 7 21:36:15.108708507 (5h9m28.695764728s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:15.108712334 (5h9m28.695768555s) [upload=5309]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:15.108714538 (5h9m28.695770759s) [upload=5309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:15.091220957 (5h10m28.678277188s) [upload=5309]: Done uploading segments... Aug 7 21:37:15.091240674 (5h10m28.678296905s) [upload=5309]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:15.177436931 (5h10m28.764493172s) [upload=5324]: Starting upload Aug 7 21:37:15.177533852 (5h10m28.764590103s) [upload=5324]: Done starting upload Aug 7 21:37:15.177602810 (5h10m28.764659061s) [upload=5324]: Uploading segments... Aug 7 21:37:15.177645340 (5h10m28.764701581s) [upload=5324]: Got next segment. Inline: true Aug 7 21:37:15.177650961 (5h10m28.764707202s) [upload=5324]: Waiting for error group managing segments... Aug 7 21:37:15.177698649 (5h10m28.76475489s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:15.177704300 (5h10m28.764760541s) [upload=5324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:15.177708177 (5h10m28.764764428s) [upload=5324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:40.730185715 (5h10m54.317241957s) [upload=5324]: Done uploading segments... Aug 7 21:37:40.730203639 (5h10m54.31725988s) [upload=5324]: Upload finished. err: <nil> Aug 7 21:38:15.287715124 (5h11m28.874771365s) [upload=5341]: Starting upload Aug 7 21:38:15.287780276 (5h11m28.874836516s) [upload=5341]: Done starting upload Aug 7 21:38:15.287844856 (5h11m28.874901107s) [upload=5341]: Uploading segments... Aug 7 21:38:15.287889961 (5h11m28.874946201s) [upload=5341]: Got next segment. Inline: true Aug 7 21:38:15.287895600 (5h11m28.874951842s) [upload=5341]: Waiting for error group managing segments... Aug 7 21:38:15.287984066 (5h11m28.875040317s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:15.287990077 (5h11m28.875046318s) [upload=5341]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:15.287993564 (5h11m28.875049804s) [upload=5341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:15.275679218 (5h12m28.862735459s) [upload=5341]: Done uploading segments... Aug 7 21:39:15.275708622 (5h12m28.862764883s) [upload=5341]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:15.358605080 (5h12m28.945661322s) [upload=5358]: Starting upload Aug 7 21:39:15.358690641 (5h12m28.945746882s) [upload=5358]: Done starting upload Aug 7 21:39:15.358748869 (5h12m28.945805111s) [upload=5358]: Uploading segments... Aug 7 21:39:15.358790938 (5h12m28.94584718s) [upload=5358]: Got next segment. Inline: true Aug 7 21:39:15.358796779 (5h12m28.94585302s) [upload=5358]: Waiting for error group managing segments... Aug 7 21:39:15.358840480 (5h12m28.945896721s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:15.358846441 (5h12m28.945902683s) [upload=5358]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:15.358849918 (5h12m28.945906159s) [upload=5358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:15.344303433 (5h13m28.931359654s) [upload=5358]: Done uploading segments... Aug 7 21:40:15.344323120 (5h13m28.931379331s) [upload=5358]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001376447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00122947s FAIL (2025-08-07 21:40:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001246842s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.....................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001543418s FAIL (2025-08-07 21:40:10 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001577311s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [...................................X........XX...X..........] duration: 24.838064264s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:40:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 24.83812666s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 406.770848ms FAIL (2025-08-07 21:40:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 406.804611ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000869009s FAIL (2025-08-07 21:39:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000884426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001481417s FAIL (2025-08-07 21:39:53 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 260]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:52.389446301 (5h7m5.976502512s) [upload=5268]: Starting upload Aug 7 21:33:52.389500824 (5h7m5.976557045s) [upload=5268]: Done starting upload Aug 7 21:33:52.389543955 (5h7m5.976600166s) [upload=5268]: Uploading segments... Aug 7 21:33:52.389575513 (5h7m5.976631724s) [upload=5268]: Got next segment. Inline: true Aug 7 21:33:52.389578869 (5h7m5.97663509s) [upload=5268]: Waiting for error group managing segments... Aug 7 21:33:52.389609948 (5h7m5.976666159s) [upload=5268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:52.389614816 (5h7m5.976671037s) [upload=5268]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:52.389617161 (5h7m5.976673382s) [upload=5268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:22.368136564 (5h7m35.955192785s) [upload=5268]: Done uploading segments... Aug 7 21:34:22.368152063 (5h7m35.955208274s) [upload=5268]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:34.745899909 (5h7m48.33295615s) [upload=5280]: Starting upload Aug 7 21:34:34.745978926 (5h7m48.333035167s) [upload=5280]: Done starting upload Aug 7 21:34:34.746045030 (5h7m48.333101271s) [upload=5280]: Uploading segments... Aug 7 21:34:34.746147852 (5h7m48.333204143s) [upload=5280]: Got next segment. Inline: true Aug 7 21:34:34.746158962 (5h7m48.333215204s) [upload=5280]: Waiting for error group managing segments... Aug 7 21:34:34.746226188 (5h7m48.333282429s) [upload=5280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:34.746232580 (5h7m48.333288821s) [upload=5280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:34.746236337 (5h7m48.333292577s) [upload=5280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:04.734023380 (5h8m18.321079591s) [upload=5280]: Done uploading segments... Aug 7 21:35:04.734039791 (5h8m18.321096012s) [upload=5280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:04.829327409 (5h8m18.41638363s) [upload=5289]: Starting upload Aug 7 21:35:04.829392180 (5h8m18.416448391s) [upload=5289]: Done starting upload Aug 7 21:35:04.829434348 (5h8m18.416490559s) [upload=5289]: Uploading segments... Aug 7 21:35:04.829462421 (5h8m18.416518642s) [upload=5289]: Got next segment. Inline: true Aug 7 21:35:04.829465987 (5h8m18.416522209s) [upload=5289]: Waiting for error group managing segments... Aug 7 21:35:04.829496094 (5h8m18.416552315s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:04.829499741 (5h8m18.416555962s) [upload=5289]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:04.829501985 (5h8m18.416558196s) [upload=5289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:34.812405766 (5h8m48.399461987s) [upload=5289]: Done uploading segments... Aug 7 21:35:34.812425613 (5h8m48.399481834s) [upload=5289]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.889033201 (5h8m48.476089413s) [upload=5299]: Starting upload Aug 7 21:35:34.889085949 (5h8m48.47614217s) [upload=5299]: Done starting upload Aug 7 21:35:34.889153155 (5h8m48.476209376s) [upload=5299]: Uploading segments... Aug 7 21:35:34.889184734 (5h8m48.476240955s) [upload=5299]: Got next segment. Inline: true Aug 7 21:35:34.889188272 (5h8m48.476244482s) [upload=5299]: Waiting for error group managing segments... Aug 7 21:35:34.889220581 (5h8m48.476276802s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.889224167 (5h8m48.476280388s) [upload=5299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.889226462 (5h8m48.476282673s) [upload=5299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.877522561 (5h9m18.464578782s) [upload=5299]: Done uploading segments... Aug 7 21:36:04.877540114 (5h9m18.464596325s) [upload=5299]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.978290179 (5h9m18.5653464s) [upload=5305]: Starting upload Aug 7 21:36:04.978343338 (5h9m18.565399559s) [upload=5305]: Done starting upload Aug 7 21:36:04.978386448 (5h9m18.565442659s) [upload=5305]: Uploading segments... Aug 7 21:36:04.978417035 (5h9m18.565473256s) [upload=5305]: Got next segment. Inline: true Aug 7 21:36:04.978420993 (5h9m18.565477214s) [upload=5305]: Waiting for error group managing segments... Aug 7 21:36:04.978450778 (5h9m18.565507s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.978454776 (5h9m18.565510997s) [upload=5305]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.978457190 (5h9m18.565513412s) [upload=5305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.966368579 (5h9m48.5534248s) [upload=5305]: Done uploading segments... Aug 7 21:36:34.966387475 (5h9m48.553443696s) [upload=5305]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:52.348809979 (5h10m5.9358662s) [upload=5319]: Starting upload Aug 7 21:36:52.348858119 (5h10m5.93591434s) [upload=5319]: Done starting upload Aug 7 21:36:52.348898024 (5h10m5.935954245s) [upload=5319]: Uploading segments... Aug 7 21:36:52.348925776 (5h10m5.935981997s) [upload=5319]: Got next segment. Inline: true Aug 7 21:36:52.348929652 (5h10m5.935985864s) [upload=5319]: Waiting for error group managing segments... Aug 7 21:36:52.348958357 (5h10m5.936014567s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:52.348963487 (5h10m5.936019697s) [upload=5319]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:52.348965901 (5h10m5.936022122s) [upload=5319]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:22.335555108 (5h10m35.922611329s) [upload=5319]: Done uploading segments... Aug 7 21:37:22.335570486 (5h10m35.922626707s) [upload=5319]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:24.787968674 (5h10m38.375024885s) [upload=5327]: Starting upload Aug 7 21:37:24.788026372 (5h10m38.375082583s) [upload=5327]: Done starting upload Aug 7 21:37:24.788072297 (5h10m38.375128518s) [upload=5327]: Uploading segments... Aug 7 21:37:24.788129475 (5h10m38.375185696s) [upload=5327]: Got next segment. Inline: true Aug 7 21:37:24.788133472 (5h10m38.375189694s) [upload=5327]: Waiting for error group managing segments... Aug 7 21:37:24.788169800 (5h10m38.375226021s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:24.788173967 (5h10m38.375230188s) [upload=5327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:24.788176171 (5h10m38.375232382s) [upload=5327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:54.773685524 (5h11m8.360741775s) [upload=5327]: Done uploading segments... Aug 7 21:37:54.773706774 (5h11m8.360763025s) [upload=5327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:56.437147115 (5h11m10.024203357s) [upload=5336]: Starting upload Aug 7 21:37:56.437251630 (5h11m10.024307872s) [upload=5336]: Done starting upload Aug 7 21:37:56.437342400 (5h11m10.024398641s) [upload=5336]: Uploading segments... Aug 7 21:37:56.437415256 (5h11m10.024471508s) [upload=5336]: Got next segment. Inline: true Aug 7 21:37:56.437425486 (5h11m10.024481767s) [upload=5336]: Waiting for error group managing segments... Aug 7 21:37:56.437531414 (5h11m10.024587665s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:56.437544558 (5h11m10.02460085s) [upload=5336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:56.437551260 (5h11m10.024607511s) [upload=5336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:26.422314839 (5h11m40.009371081s) [upload=5336]: Done uploading segments... Aug 7 21:38:26.422335869 (5h11m40.00939211s) [upload=5336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:26.513648977 (5h11m40.100705218s) [upload=5344]: Starting upload Aug 7 21:38:26.513731310 (5h11m40.100787552s) [upload=5344]: Done starting upload Aug 7 21:38:26.513792245 (5h11m40.100848496s) [upload=5344]: Uploading segments... Aug 7 21:38:26.513834112 (5h11m40.100890354s) [upload=5344]: Got next segment. Inline: true Aug 7 21:38:26.513839653 (5h11m40.100895904s) [upload=5344]: Waiting for error group managing segments... Aug 7 21:38:26.513884006 (5h11m40.100940247s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:26.513889947 (5h11m40.100946188s) [upload=5344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:26.513893434 (5h11m40.100949675s) [upload=5344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:56.502052245 (5h12m10.089108537s) [upload=5344]: Done uploading segments... Aug 7 21:38:56.502102730 (5h12m10.089158972s) [upload=5344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:24.008006172 (5h12m37.595062394s) [upload=5359]: Starting upload Aug 7 21:39:24.008059211 (5h12m37.595115422s) [upload=5359]: Done starting upload Aug 7 21:39:24.008103604 (5h12m37.595159835s) [upload=5359]: Uploading segments... Aug 7 21:39:24.008156191 (5h12m37.595212413s) [upload=5359]: Got next segment. Inline: true Aug 7 21:39:24.008162193 (5h12m37.595218414s) [upload=5359]: Waiting for error group managing segments... Aug 7 21:39:24.008207327 (5h12m37.595263548s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:24.008212668 (5h12m37.595268878s) [upload=5359]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:24.008216064 (5h12m37.595272285s) [upload=5359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:53.997226503 (5h13m7.584282744s) [upload=5359]: Done uploading segments... Aug 7 21:39:53.997270334 (5h13m7.584326586s) [upload=5359]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001501244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000704526s FAIL (2025-08-07 21:39:40 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000723112s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.................................XX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001060032s FAIL (2025-08-07 21:39:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001076974s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................................X........XX...X.........] duration: 1m31.812626238s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:39:38 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1m31.812660151s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000411101s FAIL (2025-08-07 21:39:35 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 7 21:30:41.839223772 (5h3m55.426280023s) [upload=5217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:41.839229273 (5h3m55.426285513s) [upload=5217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:11.826385247 (5h4m25.413441468s) [upload=5217]: Next segment err: context deadline exceeded Aug 7 21:31:11.826544515 (5h4m25.413600736s) [upload=5217]: Done uploading segments... Aug 7 21:31:11.826568178 (5h4m25.413624399s) [upload=5217]: Upload finished. err: context deadline exceeded Aug 7 21:31:11.924007243 (5h4m25.511063484s) [upload=5227]: Starting upload Aug 7 21:31:11.924116868 (5h4m25.511173108s) [upload=5227]: Done starting upload Aug 7 21:31:11.924293237 (5h4m25.511349458s) [upload=5227]: Uploading segments... Aug 7 21:31:11.924515282 (5h4m25.511571553s) [upload=5227]: Got next segment. Inline: false Aug 7 21:31:11.924580593 (5h4m25.511636844s) [upload=5227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.924591073 (5h4m25.511647324s) [upload=5227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:41.901407105 (5h4m55.488463316s) [upload=5227]: Next segment err: context deadline exceeded Aug 7 21:31:41.901545804 (5h4m55.488602015s) [upload=5227]: Done uploading segments... Aug 7 21:31:41.901568547 (5h4m55.488624747s) [upload=5227]: Upload finished. err: context deadline exceeded Aug 7 21:32:25.036046257 (5h5m38.623102498s) [upload=5243]: Starting upload Aug 7 21:32:25.036157154 (5h5m38.623213395s) [upload=5243]: Done starting upload Aug 7 21:32:25.036227865 (5h5m38.623284077s) [upload=5243]: Uploading segments... Aug 7 21:32:25.036538426 (5h5m38.623594667s) [upload=5243]: Got next segment. Inline: false Aug 7 21:32:25.036580995 (5h5m38.623637236s) [upload=5243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:25.036586435 (5h5m38.623642676s) [upload=5243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:55.024878175 (5h6m8.611934396s) [upload=5243]: Next segment err: context deadline exceeded Aug 7 21:32:55.024926696 (5h6m8.611982907s) [upload=5243]: Done uploading segments... Aug 7 21:32:55.024945942 (5h6m8.612002163s) [upload=5243]: Upload finished. err: context deadline exceeded Aug 7 21:32:55.115854025 (5h6m8.702910266s) [upload=5248]: Starting upload Aug 7 21:32:55.116010426 (5h6m8.703066667s) [upload=5248]: Done starting upload Aug 7 21:32:55.116150518 (5h6m8.703206779s) [upload=5248]: Uploading segments... Aug 7 21:32:55.116485283 (5h6m8.703541534s) [upload=5248]: Got next segment. Inline: false Aug 7 21:32:55.116547149 (5h6m8.7036034s) [upload=5248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:55.116554834 (5h6m8.703611074s) [upload=5248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:25.100546070 (5h6m38.687602282s) [upload=5248]: Next segment err: context deadline exceeded Aug 7 21:33:25.100697263 (5h6m38.687753474s) [upload=5248]: Done uploading segments... Aug 7 21:33:25.100715427 (5h6m38.687771648s) [upload=5248]: Upload finished. err: context deadline exceeded Aug 7 21:33:25.196632841 (5h6m38.783689062s) [upload=5261]: Starting upload Aug 7 21:33:25.196680791 (5h6m38.783737002s) [upload=5261]: Done starting upload Aug 7 21:33:25.196737997 (5h6m38.783794208s) [upload=5261]: Uploading segments... Aug 7 21:33:25.196895151 (5h6m38.783951362s) [upload=5261]: Got next segment. Inline: false Aug 7 21:33:25.196920908 (5h6m38.78397712s) [upload=5261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.196924515 (5h6m38.783980726s) [upload=5261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:55.184515726 (5h7m8.771571978s) [upload=5261]: Next segment err: context deadline exceeded Aug 7 21:33:55.184605584 (5h7m8.771661825s) [upload=5261]: Done uploading segments... Aug 7 21:33:55.184627856 (5h7m8.771684107s) [upload=5261]: Upload finished. err: context deadline exceeded Aug 7 21:33:55.280831383 (5h7m8.867887624s) [upload=5269]: Starting upload Aug 7 21:33:55.280916101 (5h7m8.867972351s) [upload=5269]: Done starting upload Aug 7 21:33:55.281021477 (5h7m8.868077719s) [upload=5269]: Uploading segments... Aug 7 21:33:55.281335644 (5h7m8.868391884s) [upload=5269]: Got next segment. Inline: false Aug 7 21:33:55.281394694 (5h7m8.868450945s) [upload=5269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:55.281400585 (5h7m8.868456827s) [upload=5269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:25.266191309 (5h7m38.85324752s) [upload=5269]: Next segment err: context deadline exceeded Aug 7 21:34:25.266301575 (5h7m38.853357796s) [upload=5269]: Done uploading segments... Aug 7 21:34:25.266319499 (5h7m38.85337572s) [upload=5269]: Upload finished. err: context deadline exceeded Aug 7 21:34:29.252320253 (5h7m42.839376495s) [upload=5279]: Starting upload Aug 7 21:34:29.252400804 (5h7m42.839457045s) [upload=5279]: Done starting upload Aug 7 21:34:29.252468230 (5h7m42.839524471s) [upload=5279]: Uploading segments... Aug 7 21:34:29.252769131 (5h7m42.839825383s) [upload=5279]: Got next segment. Inline: false Aug 7 21:34:29.252812733 (5h7m42.839868975s) [upload=5279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:29.252817773 (5h7m42.839874014s) [upload=5279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:59.239490102 (5h8m12.826546323s) [upload=5279]: Next segment err: context deadline exceeded Aug 7 21:34:59.239565343 (5h8m12.826621554s) [upload=5279]: Done uploading segments... Aug 7 21:34:59.239579939 (5h8m12.826636151s) [upload=5279]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.372769473 (5h8m32.959825724s) [upload=5296]: Starting upload Aug 7 21:35:19.372862917 (5h8m32.959919168s) [upload=5296]: Done starting upload Aug 7 21:35:19.373171613 (5h8m32.960227864s) [upload=5296]: Uploading segments... Aug 7 21:35:19.373426359 (5h8m32.96048261s) [upload=5296]: Got next segment. Inline: false Aug 7 21:35:19.373476772 (5h8m32.960533014s) [upload=5296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.373485549 (5h8m32.96054179s) [upload=5296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:49.354206855 (5h9m2.941263097s) [upload=5296]: Next segment err: context deadline exceeded Aug 7 21:35:49.354281264 (5h9m2.941337505s) [upload=5296]: Done uploading segments... Aug 7 21:35:49.354300139 (5h9m2.94135638s) [upload=5296]: Upload finished. err: context deadline exceeded Aug 7 21:36:27.376191725 (5h9m40.963247946s) [upload=5313]: Starting upload Aug 7 21:36:27.376242419 (5h9m40.96329865s) [upload=5313]: Done starting upload Aug 7 21:36:27.376347896 (5h9m40.963404106s) [upload=5313]: Uploading segments... Aug 7 21:36:27.376511180 (5h9m40.963567401s) [upload=5313]: Got next segment. Inline: false Aug 7 21:36:27.376539804 (5h9m40.963596025s) [upload=5313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:27.376544233 (5h9m40.963600454s) [upload=5313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:57.366652335 (5h10m10.953708576s) [upload=5313]: Next segment err: context deadline exceeded Aug 7 21:36:57.366728869 (5h10m10.95378511s) [upload=5313]: Done uploading segments... Aug 7 21:36:57.366751832 (5h10m10.953808073s) [upload=5313]: Upload finished. err: context deadline exceeded Aug 7 21:36:57.461008885 (5h10m11.048065106s) [upload=5321]: Starting upload Aug 7 21:36:57.461087872 (5h10m11.048144093s) [upload=5321]: Done starting upload Aug 7 21:36:57.461230760 (5h10m11.048287s) [upload=5321]: Uploading segments... Aug 7 21:36:57.461340454 (5h10m11.048396675s) [upload=5321]: Got next segment. Inline: false Aug 7 21:36:57.461375219 (5h10m11.048431429s) [upload=5321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:57.461378755 (5h10m11.048434966s) [upload=5321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:27.449244931 (5h10m41.036301182s) [upload=5321]: Next segment err: context deadline exceeded Aug 7 21:37:27.449424726 (5h10m41.036480967s) [upload=5321]: Done uploading segments... Aug 7 21:37:27.449465262 (5h10m41.036521503s) [upload=5321]: Upload finished. err: context deadline exceeded Aug 7 21:37:55.501043683 (5h11m9.088099934s) [upload=5335]: Starting upload Aug 7 21:37:55.501146705 (5h11m9.088202946s) [upload=5335]: Done starting upload Aug 7 21:37:55.501210003 (5h11m9.088266245s) [upload=5335]: Uploading segments... Aug 7 21:37:55.501506116 (5h11m9.088562357s) [upload=5335]: Got next segment. Inline: false Aug 7 21:37:55.501546672 (5h11m9.088602913s) [upload=5335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:55.501552343 (5h11m9.088608584s) [upload=5335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:25.489167397 (5h11m39.076223638s) [upload=5335]: Next segment err: context deadline exceeded Aug 7 21:38:25.489236585 (5h11m39.076292827s) [upload=5335]: Done uploading segments... Aug 7 21:38:25.489255791 (5h11m39.076312033s) [upload=5335]: Upload finished. err: context deadline exceeded Aug 7 21:39:05.437960161 (5h12m19.025016382s) [upload=5353]: Starting upload Aug 7 21:39:05.438020904 (5h12m19.025077125s) [upload=5353]: Done starting upload Aug 7 21:39:05.438137612 (5h12m19.025193863s) [upload=5353]: Uploading segments... Aug 7 21:39:05.438275971 (5h12m19.025332192s) [upload=5353]: Got next segment. Inline: false Aug 7 21:39:05.438348736 (5h12m19.025404957s) [upload=5353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:05.438353515 (5h12m19.025409736s) [upload=5353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:35.425840953 (5h12m49.012897194s) [upload=5353]: Next segment err: context deadline exceeded Aug 7 21:39:35.425899322 (5h12m49.012955573s) [upload=5353]: Done uploading segments... Aug 7 21:39:35.425916785 (5h12m49.012973036s) [upload=5353]: Upload finished. err: context deadline exceeded Duration: 30.000425308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.000948867s FAIL (2025-08-07 21:39:34 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 40 (5h10m17.413544981s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.826841057 (5h10m17.413897299s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.826983933 (5h10m17.414040174s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.827218521 (5h10m17.414274763s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.827383179 (5h10m17.41443942s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.827627445 (5h10m17.414683687s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.827751647 (5h10m17.414807898s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.828066434 (5h10m17.415122676s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.828167473 (5h10m17.415223714s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.828451093 (5h10m17.415507334s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.828515743 (5h10m17.415571985s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Uploading piece... Aug 7 21:37:03.934711994 (5h10m17.521768225s) [upload=5311]: Next returned nil segment Aug 7 21:37:03.934726611 (5h10m17.521782853s) [upload=5311]: Waiting for error group managing segments... Aug 7 21:37:03.953545219 (5h10m17.540601461s) [noise=true , address=45.91.116.6:4996 , piece_id=RCONEEDI4GDVROD64WTUAH3REURQSENYKRP3GXRGY53AIM2XV5MQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.958188677 (5h10m17.545244919s) [noise=true , address=104.36.82.39:4746 , piece_id=KJEXRIFPMBIWSULKGZOSUDIFO5JSLOW24QDPHNKBVP24BV4HG6SA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.964982629 (5h10m17.552038869s) [noise=true , address=104.36.82.61:4380 , piece_id=EYGHCC2VBWOBOHPWYWA2CEV7JVI5QYAVDHWP3NGF6SYMMGPTQFDQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.969165096 (5h10m17.556221337s) [noise=true , address=104.36.82.19:4426 , piece_id=3QBNLWLB2C4ACZTZ276C4VHGPUYX7NWV3YWE3YJMSCCOUL442RHA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.973698649 (5h10m17.56075489s) [noise=true , address=104.36.82.53:4141 , piece_id=ED5MIJF6D2MOWFMGQ46XDSPTXVBEAYIBV3CSZEQ36LELLPJ6WZFQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.975101527 (5h10m17.562157768s) [noise=true , address=45.91.116.14:4213 , piece_id=HOJLN6PQSGEE2V3W4YUEOKK4C6MU2E64XPIDUQIVPEO7WYPQBITA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.975868810 (5h10m17.562925051s) [noise=true , address=104.36.82.8:4999 , piece_id=NV4OH27A53IKNMG6A47H4QQZR3UKEKAJWRJHEJN5SOSSIKRHOK2A, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.976647924 (5h10m17.563704165s) [noise=true , address=104.36.82.54:4164 , piece_id=ITHJPEROUZJPDFXZJMEMLSMTP352C7JZBG66QA467AOD3KU6ARVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.981871894 (5h10m17.568928146s) [noise=true , address=45.91.116.7:4030 , piece_id=POLX3ZADAQDDS3EABO2RCGTAT4Z23FY7MSSNXM7TWYYSWF4HFEYQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.990165577 (5h10m17.577221818s) [noise=true , address=104.36.82.39:4741 , piece_id=NSVMWM5EFGB6J4UXAWVJZM2BQNNSZQHE6JA2ES4PYLHACVVJQOWQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:03.996463181 (5h10m17.583519422s) [noise=true , address=77.90.31.16:4742 , piece_id=ERUGO4MG35TTSV76XZAL4DL76A5C3XQBGOT4SE3B3KYCNQLDGIXA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:04.067745290 (5h10m17.65480153s) [noise=true , address=104.36.82.47:4545 , piece_id=JI4XEDNFM2C3XUT5JD2Q5OFBN5EW77GVKKNB75BSZML24VIZNRNQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:04.194336421 (5h10m17.781392672s) [noise=true , address=104.36.82.43:4849 , piece_id=BOX37GRJHMNRG4SVZ6WNCKUHS5ASA7AZGXXM73JBHCI5LMOSE7QA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:04.231214259 (5h10m17.8182705s) [noise=true , address=104.36.82.45:4930 , piece_id=KBE5SBD7GGJNACM7RJFNV2LEENAUZ2LKRM5CFBFQ75GZXZFW2KWA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: <nil> Aug 7 21:37:04.231245878 (5h10m17.818302118s) [seg_pos={0 0}, upload=5311]: Segment reached optimal threshold of 14 pieces. Aug 7 21:37:04.231458835 (5h10m17.818515086s) [noise=true , address=79.127.205.234:4600 , piece_id=V3ZRDN77DCY4ZPYZJLIAWT76WGDUQHE4BAK23RH5D7DCEX5M3FVA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: expected piece hash; context canceled; EOF Aug 7 21:37:04.231571595 (5h10m17.818627836s) [noise=true , address=104.36.82.23:4472 , piece_id=G45VA52TYEJYQU36RRE4H67F5JKFA63A6ZG67PU5KMCYTXAMZUUQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: read tcp 5.161.89.252:51344->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:04.231596983 (5h10m17.818653224s) [noise=true , address=64.49.13.155:4914 , piece_id=6AUGLHYNHGAKMNJRMNA74PJFKMCCCSKHK3ETQICHKCUGWFTWTCMQ, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123exKWZgTdDY8ojhiFcaXkAQnsVs2MZp5MWfcaPiwk5f5tFR9A): protocol: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64916->64.49.13.155:4914: use of closed network connection Aug 7 21:37:04.231638852 (5h10m17.818695092s) [noise=true , address=104.36.82.58:4287 , piece_id=ETEE347VQKLH2CZUDKNWEE3CDMSKQHMZ3NJGXUL7FL54IYPDFICA, seg_pos={0 0}, upload=5311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wNKharWFFSSsLMd1vZYi6zYEYBe3VT6Z3ikrY7Cjo3xPLUK62): protocol: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:50990->104.36.82.58:4287: use of closed network connection Aug 7 21:37:04.231659821 (5h10m17.818716062s) [seg_pos={0 0}, upload=5311]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:37:04.231740732 (5h10m17.818796972s) [upload=5311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:37:04.231746432 (5h10m17.818802674s) [upload=5311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:34.607598692 (5h10m48.194654932s) [upload=5311]: Done uploading segments... Aug 7 21:37:34.607648003 (5h10m48.194704254s) [upload=5311]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:34.704354810 (5h10m48.291411021s) [upload=5330]: Starting upload Aug 7 21:37:34.704409994 (5h10m48.291466215s) [upload=5330]: Done starting upload Aug 7 21:37:34.704465938 (5h10m48.291522179s) [upload=5330]: Uploading segments... Aug 7 21:37:34.704648018 (5h10m48.291704229s) [upload=5330]: Got next segment. Inline: false Aug 7 21:37:34.704676862 (5h10m48.291733083s) [upload=5330]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.704680678 (5h10m48.291736899s) [upload=5330]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:34.684727743 (5h11m48.271783975s) [upload=5330]: Next segment err: context deadline exceeded Aug 7 21:38:34.684789619 (5h11m48.27184584s) [upload=5330]: Done uploading segments... Aug 7 21:38:34.684808654 (5h11m48.271864865s) [upload=5330]: Upload finished. err: context deadline exceeded Aug 7 21:38:34.770676549 (5h11m48.357732759s) [upload=5346]: Starting upload Aug 7 21:38:34.770726071 (5h11m48.357782283s) [upload=5346]: Done starting upload Aug 7 21:38:34.770764212 (5h11m48.357820423s) [upload=5346]: Uploading segments... Aug 7 21:38:34.770936515 (5h11m48.357992736s) [upload=5346]: Got next segment. Inline: false Aug 7 21:38:34.770964016 (5h11m48.358020237s) [upload=5346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:34.770967763 (5h11m48.358023984s) [upload=5346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:34.756776177 (5h12m48.343832417s) [upload=5346]: Next segment err: context deadline exceeded Aug 7 21:39:34.756912992 (5h12m48.343969233s) [upload=5346]: Done uploading segments... Aug 7 21:39:34.756934332 (5h12m48.343990574s) [upload=5346]: Upload finished. err: context deadline exceeded Duration: 1m0.000962723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000576023s FAIL (2025-08-07 21:39:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000596741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 993.808548ms FAIL (2025-08-07 21:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 993.834577ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)