Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXX.X..................................XXXXXXXXXXXXXXXX.X] duration: 30.000383824s FAIL (2025-08-07 22:24:48 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00040331s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 110.102141ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:24:26 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 110.12223ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 170.031579ms FAIL (2025-08-07 22:24:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 170.059321ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.........................................XXXXXXXXXXXXXXXXX] duration: 30.000842754s FAIL (2025-08-07 22:24:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000852059s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX..................................XXXXX] duration: 30.001590094s FAIL (2025-08-07 22:24:22 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001607838s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X........................................XXXXXXXXXXXXXX.X] duration: 30.001310743s FAIL (2025-08-07 22:24:17 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== finished. err: context deadline exceeded Aug 7 22:22:59.728546153 (5h56m13.315602364s) [upload=6077]: Starting upload Aug 7 22:22:59.728603158 (5h56m13.31565937s) [upload=6077]: Done starting upload Aug 7 22:22:59.728642352 (5h56m13.315698573s) [upload=6077]: Uploading segments... Aug 7 22:22:59.728924689 (5h56m13.315980899s) [upload=6077]: Got next segment. Inline: false Aug 7 22:22:59.728958091 (5h56m13.316014302s) [upload=6077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.728963792 (5h56m13.316020013s) [upload=6077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:22:59.747162121 (5h56m13.334218342s) [seg_pos={0 0}, upload=6077]: Begin upload segment... Aug 7 22:22:59.747199562 (5h56m13.334255783s) [seg_pos={0 0}, upload=6077]: Done begin upload segment. Aug 7 22:22:59.747223407 (5h56m13.334279628s) [noise=true , address=45.91.116.7:4011 , piece_id=E7YGVXQXIO2A34B22AHGKBQXER4KLEX3AFGHVFUQ6USLQMUFSW2Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.747566337 (5h56m13.334622558s) [noise=true , address=104.36.82.11:4115 , piece_id=BALIRX6S5WW2CGSXSIFHWV2ABDH3ASPHR36YEYRUYTTGZCW6U6AQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.747878128 (5h56m13.334934349s) [noise=true , address=77.90.31.19:4842 , piece_id=ULCHXICRO3ULX7IEW6GVXHHGXT6F42N3WQDEEUVOSKCXID6F6XZQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.748271662 (5h56m13.335327883s) [noise=true , address=45.91.116.12:4225 , piece_id=MNJGBICBIG3HQXCO7B2C2ITRFSOW475RAGNVTAQNVFQYAJ7DOU5Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.748638838 (5h56m13.335695079s) [noise=true , address=104.36.82.10:4079 , piece_id=DHLJEMGMQUTVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749054955 (5h56m13.336111176s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749426248 (5h56m13.336482459s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.750681361 (5h56m13.337737582s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751093369 (5h56m13.338149581s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751643326 (5h56m13.338699547s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752249137 (5h56m13.339305348s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752826465 (5h56m13.339882676s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753473372 (5h56m13.340529583s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753961884 (5h56m13.341018096s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.754616727 (5h56m13.341672948s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.755122451 (5h56m13.342178672s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.837536198 (5h56m13.424592419s) [upload=6077]: Next returned nil segment Aug 7 22:22:59.837547418 (5h56m13.424603629s) [upload=6077]: Waiting for error group managing segments... Aug 7 22:22:59.846810959 (5h56m13.43386718s) [noise=true , address=45.91.116.7:4011 , piece_id=E7YGVXQXIO2A34B22AHGKBQXER4KLEX3AFGHVFUQ6USLQMUFSW2Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.847279324 (5h56m13.434335535s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.848559454 (5h56m13.435615675s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852382881 (5h56m13.439439102s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852577974 (5h56m13.439634195s) [noise=true , address=77.90.31.19:4842 , piece_id=ULCHXICRO3ULX7IEW6GVXHHGXT6F42N3WQDEEUVOSKCXID6F6XZQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856642091 (5h56m13.443698302s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856766534 (5h56m13.443822745s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868085922 (5h56m13.455142143s) [noise=true , address=45.91.116.12:4225 , piece_id=MNJGBICBIG3HQXCO7B2C2ITRFSOW475RAGNVTAQNVFQYAJ7DOU5Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868246762 (5h56m13.455302973s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.877804453 (5h56m13.464860674s) [noise=true , address=104.36.82.11:4115 , piece_id=BALIRX6S5WW2CGSXSIFHWV2ABDH3ASPHR36YEYRUYTTGZCW6U6AQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.882357252 (5h56m13.469413462s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887398722 (5h56m13.474454933s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887674137 (5h56m13.474730358s) [noise=true , address=104.36.82.10:4079 , piece_id=DHLJEMGMQUTVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887689665 (5h56m13.474745877s) [seg_pos={0 0}, upload=6077]: Segment reached optimal threshold of 13 pieces. Aug 7 22:22:59.887775385 (5h56m13.474831606s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887795453 (5h56m13.474851674s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887879469 (5h56m13.47493569s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection Aug 7 22:22:59.887943669 (5h56m13.47499988s) [seg_pos={0 0}, upload=6077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:22:59.888006888 (5h56m13.475063109s) [upload=6077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:22:59.888010665 (5h56m13.475066886s) [upload=6077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.933305033 (5h56m13.520361243s) [upload=6077]: Done uploading segments... Aug 7 22:22:59.933317746 (5h56m13.520373958s) [upload=6077]: Upload finished. err: <nil> Aug 7 22:23:47.739190890 (5h57m1.326247131s) [upload=6091]: Starting upload Aug 7 22:23:47.739297178 (5h57m1.326353419s) [upload=6091]: Done starting upload Aug 7 22:23:47.739428053 (5h57m1.326484274s) [upload=6091]: Uploading segments... Aug 7 22:23:47.739777785 (5h57m1.326834025s) [upload=6091]: Got next segment. Inline: false Aug 7 22:23:47.739805196 (5h57m1.326861407s) [upload=6091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:47.739811638 (5h57m1.326867849s) [upload=6091]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:17.722308743 (5h57m31.309364985s) [upload=6091]: Next segment err: context deadline exceeded Aug 7 22:24:17.722438996 (5h57m31.309495238s) [upload=6091]: Done uploading segments... Aug 7 22:24:17.722476005 (5h57m31.309532246s) [upload=6091]: Upload finished. err: context deadline exceeded Duration: 30.001327995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X.XX...............................XXXXXXXXXXX.] duration: 513.854599ms FAIL (2025-08-07 22:24:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:04.228585182 (5h50m17.815641403s) [upload=5990]: Starting upload Aug 7 22:17:04.228636998 (5h50m17.815693219s) [upload=5990]: Done starting upload Aug 7 22:17:04.228690769 (5h50m17.81574699s) [upload=5990]: Uploading segments... Aug 7 22:17:04.228721386 (5h50m17.815777607s) [upload=5990]: Got next segment. Inline: true Aug 7 22:17:04.228725082 (5h50m17.815781303s) [upload=5990]: Waiting for error group managing segments... Aug 7 22:17:04.228765939 (5h50m17.815822161s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:04.228772071 (5h50m17.815828292s) [upload=5990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:04.228775867 (5h50m17.815832089s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:04.215607888 (5h51m17.802664109s) [upload=5990]: Done uploading segments... Aug 7 22:18:04.215625421 (5h51m17.802681632s) [upload=5990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:04.292132983 (5h51m17.879189234s) [upload=6004]: Starting upload Aug 7 22:18:04.292228702 (5h51m17.879284943s) [upload=6004]: Done starting upload Aug 7 22:18:04.292318189 (5h51m17.879374439s) [upload=6004]: Uploading segments... Aug 7 22:18:04.292362020 (5h51m17.879418262s) [upload=6004]: Got next segment. Inline: true Aug 7 22:18:04.292367411 (5h51m17.879423662s) [upload=6004]: Waiting for error group managing segments... Aug 7 22:18:04.292410401 (5h51m17.879466651s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:04.292416432 (5h51m17.879472673s) [upload=6004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:04.292419948 (5h51m17.8794762s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:04.275395564 (5h52m17.862451785s) [upload=6004]: Done uploading segments... Aug 7 22:19:04.275413478 (5h52m17.862469689s) [upload=6004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:12.175471355 (5h56m25.762527616s) [upload=6082]: Starting upload Aug 7 22:23:12.175573175 (5h56m25.762629466s) [upload=6082]: Done starting upload Aug 7 22:23:12.175670356 (5h56m25.762726627s) [upload=6082]: Uploading segments... Aug 7 22:23:12.175760093 (5h56m25.762816364s) [upload=6082]: Got next segment. Inline: true Aug 7 22:23:12.175767677 (5h56m25.762823919s) [upload=6082]: Waiting for error group managing segments... Aug 7 22:23:12.175819345 (5h56m25.762875585s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:12.175825105 (5h56m25.762881346s) [upload=6082]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:12.175828732 (5h56m25.762884973s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.154559214 (5h57m25.741615455s) [upload=6082]: Done uploading segments... Aug 7 22:24:12.154578851 (5h57m25.741635092s) [upload=6082]: Upload finished. err: metaclient: context deadline exceeded Duration: 513.872904ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 169.658661ms FAIL (2025-08-07 22:24:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 169.680943ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXX...X.....X..............................XXXXXXXXXXXXXXX] duration: 30.000757235s FAIL (2025-08-07 22:24:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000774799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX.............................XXXXXXXXXXXXXXXX.] duration: 219.897159ms FAIL (2025-08-07 22:23:58 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ed Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:41.748643128 (5h50m55.33569938s) [upload=5998]: Starting upload Aug 7 22:17:41.748800843 (5h50m55.335857084s) [upload=5998]: Done starting upload Aug 7 22:17:41.748866916 (5h50m55.335923157s) [upload=5998]: Uploading segments... Aug 7 22:17:41.748948899 (5h50m55.33600514s) [upload=5998]: Got next segment. Inline: true Aug 7 22:17:41.748956914 (5h50m55.336013155s) [upload=5998]: Waiting for error group managing segments... Aug 7 22:17:41.749006376 (5h50m55.336062618s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:41.749012397 (5h50m55.336068639s) [upload=5998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:41.749016225 (5h50m55.336072466s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:11.736773915 (5h51m25.323830136s) [upload=5998]: Done uploading segments... Aug 7 22:18:11.736788051 (5h51m25.323844272s) [upload=5998]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:48.986206344 (5h52m2.573262556s) [upload=6015]: Starting upload Aug 7 22:18:48.986264964 (5h52m2.573321176s) [upload=6015]: Done starting upload Aug 7 22:18:48.986302574 (5h52m2.573358785s) [upload=6015]: Uploading segments... Aug 7 22:18:48.986327040 (5h52m2.573383251s) [upload=6015]: Got next segment. Inline: true Aug 7 22:18:48.986330306 (5h52m2.573386517s) [upload=6015]: Waiting for error group managing segments... Aug 7 22:18:48.986358459 (5h52m2.57341467s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:48.986361795 (5h52m2.573418006s) [upload=6015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:48.986364019 (5h52m2.573420231s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:18.972022741 (5h52m32.559078983s) [upload=6015]: Done uploading segments... Aug 7 22:19:18.972044513 (5h52m32.559100753s) [upload=6015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:33.193306109 (5h52m46.78036237s) [upload=6027]: Starting upload Aug 7 22:19:33.193429379 (5h52m46.78048563s) [upload=6027]: Done starting upload Aug 7 22:19:33.193516863 (5h52m46.780573103s) [upload=6027]: Uploading segments... Aug 7 22:19:33.193610827 (5h52m46.780667099s) [upload=6027]: Got next segment. Inline: true Aug 7 22:19:33.193618262 (5h52m46.780674513s) [upload=6027]: Waiting for error group managing segments... Aug 7 22:19:33.193668956 (5h52m46.780725196s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:33.193675307 (5h52m46.780731558s) [upload=6027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:33.193678734 (5h52m46.780734975s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:03.177682259 (5h53m16.76473848s) [upload=6027]: Done uploading segments... Aug 7 22:20:03.177699481 (5h53m16.764755702s) [upload=6027]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:33.558255550 (5h53m47.145311761s) [upload=6040]: Starting upload Aug 7 22:20:33.558305072 (5h53m47.145361293s) [upload=6040]: Done starting upload Aug 7 22:20:33.558352641 (5h53m47.145408852s) [upload=6040]: Uploading segments... Aug 7 22:20:33.558381975 (5h53m47.145438186s) [upload=6040]: Got next segment. Inline: true Aug 7 22:20:33.558385782 (5h53m47.145441993s) [upload=6040]: Waiting for error group managing segments... Aug 7 22:20:33.558417922 (5h53m47.145474134s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:33.558421789 (5h53m47.14547801s) [upload=6040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:33.558424405 (5h53m47.145480615s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:03.541933746 (5h54m17.128989987s) [upload=6040]: Done uploading segments... Aug 7 22:21:03.541952551 (5h54m17.129008792s) [upload=6040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:36.389854096 (5h54m49.976910308s) [upload=6055]: Starting upload Aug 7 22:21:36.389940908 (5h54m49.976997159s) [upload=6055]: Done starting upload Aug 7 22:21:36.390006921 (5h54m49.977063142s) [upload=6055]: Uploading segments... Aug 7 22:21:36.390051064 (5h54m49.977107285s) [upload=6055]: Got next segment. Inline: true Aug 7 22:21:36.390056905 (5h54m49.977113126s) [upload=6055]: Waiting for error group managing segments... Aug 7 22:21:36.390101748 (5h54m49.977157969s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:36.390107970 (5h54m49.977164191s) [upload=6055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:36.390112067 (5h54m49.977168288s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:06.377133119 (5h55m19.96418934s) [upload=6055]: Done uploading segments... Aug 7 22:22:06.377148799 (5h55m19.96420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Duration: 219.917818ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 227.60758ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:23:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 227.625634ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.........................................XXXXXXXXXXXXXXXX] duration: 30.000285233s FAIL (2025-08-07 22:23:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00030508s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXX....................................XXXXXXXXXXXX] duration: 1m0.000790979s FAIL (2025-08-07 22:23:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000803383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX..................................XXXX] duration: 30.000451077s FAIL (2025-08-07 22:23:45 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000467838s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXX.XX......X.............................XXXXXXXXXXXX] duration: 1m0.001007893s FAIL (2025-08-07 22:23:43 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== _ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Aug 7 22:16:22.269422973 (5h49m35.856479254s) [upload=5981]: Starting upload Aug 7 22:16:22.269535553 (5h49m35.856591794s) [upload=5981]: Done starting upload Aug 7 22:16:22.269676246 (5h49m35.856732487s) [upload=5981]: Uploading segments... Aug 7 22:16:22.269948395 (5h49m35.857004635s) [upload=5981]: Got next segment. Inline: false Aug 7 22:16:22.269993167 (5h49m35.857049419s) [upload=5981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:22.269999790 (5h49m35.857056041s) [upload=5981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:22.257287873 (5h50m35.844344113s) [upload=5981]: Next segment err: context deadline exceeded Aug 7 22:17:22.257515257 (5h50m35.844571508s) [upload=5981]: Done uploading segments... Aug 7 22:17:22.257574649 (5h50m35.844630889s) [upload=5981]: Upload finished. err: context deadline exceeded Aug 7 22:17:22.352562567 (5h50m35.939618808s) [upload=5994]: Starting upload Aug 7 22:17:22.352642467 (5h50m35.939698707s) [upload=5994]: Done starting upload Aug 7 22:17:22.352739408 (5h50m35.939795648s) [upload=5994]: Uploading segments... Aug 7 22:17:22.353012627 (5h50m35.940068868s) [upload=5994]: Got next segment. Inline: false Aug 7 22:17:22.353063573 (5h50m35.940119824s) [upload=5994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:22.353069433 (5h50m35.940125675s) [upload=5994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:22.334060789 (5h51m35.92111701s) [upload=5994]: Next segment err: context deadline exceeded Aug 7 22:18:22.334109780 (5h51m35.921165992s) [upload=5994]: Done uploading segments... Aug 7 22:18:22.334124208 (5h51m35.921180419s) [upload=5994]: Upload finished. err: context deadline exceeded Aug 7 22:18:22.417801973 (5h51m36.004858194s) [upload=6010]: Starting upload Aug 7 22:18:22.417851666 (5h51m36.004907878s) [upload=6010]: Done starting upload Aug 7 22:18:22.417922909 (5h51m36.00497913s) [upload=6010]: Uploading segments... Aug 7 22:18:22.418063011 (5h51m36.005119222s) [upload=6010]: Got next segment. Inline: false Aug 7 22:18:22.418083209 (5h51m36.00513943s) [upload=6010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:22.418088378 (5h51m36.005144599s) [upload=6010]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:22.407229763 (5h52m35.994286003s) [upload=6010]: Next segment err: context deadline exceeded Aug 7 22:19:22.407403607 (5h52m35.994459847s) [upload=6010]: Done uploading segments... Aug 7 22:19:22.407430978 (5h52m35.994487219s) [upload=6010]: Upload finished. err: context deadline exceeded Aug 7 22:19:22.483258289 (5h52m36.07031451s) [upload=6023]: Starting upload Aug 7 22:19:22.483313302 (5h52m36.070369514s) [upload=6023]: Done starting upload Aug 7 22:19:22.483363717 (5h52m36.070419968s) [upload=6023]: Uploading segments... Aug 7 22:19:22.483555084 (5h52m36.070611295s) [upload=6023]: Got next segment. Inline: false Aug 7 22:19:22.483607642 (5h52m36.070663852s) [upload=6023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:22.483613111 (5h52m36.070669332s) [upload=6023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:22.470951760 (5h53m36.058008001s) [upload=6023]: Next segment err: context deadline exceeded Aug 7 22:20:22.471159418 (5h53m36.058215659s) [upload=6023]: Done uploading segments... Aug 7 22:20:22.471229579 (5h53m36.058285831s) [upload=6023]: Upload finished. err: context deadline exceeded Aug 7 22:20:31.584716452 (5h53m45.171772692s) [upload=6039]: Starting upload Aug 7 22:20:31.584819494 (5h53m45.171875734s) [upload=6039]: Done starting upload Aug 7 22:20:31.584946481 (5h53m45.172002722s) [upload=6039]: Uploading segments... Aug 7 22:20:31.585228928 (5h53m45.172285168s) [upload=6039]: Got next segment. Inline: false Aug 7 22:20:31.585296344 (5h53m45.172352585s) [upload=6039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:31.585305191 (5h53m45.172361432s) [upload=6039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:31.564329759 (5h54m45.15138597s) [upload=6039]: Next segment err: context deadline exceeded Aug 7 22:21:31.564470662 (5h54m45.151526883s) [upload=6039]: Done uploading segments... Aug 7 22:21:31.564486933 (5h54m45.151543144s) [upload=6039]: Upload finished. err: context deadline exceeded Aug 7 22:21:31.630894751 (5h54m45.217950972s) [upload=6054]: Starting upload Aug 7 22:21:31.630942149 (5h54m45.21799837s) [upload=6054]: Done starting upload Aug 7 22:21:31.631165055 (5h54m45.218221276s) [upload=6054]: Uploading segments... Aug 7 22:21:31.631364788 (5h54m45.218421009s) [upload=6054]: Got next segment. Inline: false Aug 7 22:21:31.631413229 (5h54m45.21846945s) [upload=6054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:31.631419942 (5h54m45.218476182s) [upload=6054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:22:31.618721098 (5h55m45.205777349s) [upload=6054]: Next segment err: context deadline exceeded Aug 7 22:22:31.618829049 (5h55m45.20588529s) [upload=6054]: Done uploading segments... Aug 7 22:22:31.618858314 (5h55m45.205914555s) [upload=6054]: Upload finished. err: context deadline exceeded Aug 7 22:22:43.895514583 (5h55m57.482570825s) [upload=6073]: Starting upload Aug 7 22:22:43.895588541 (5h55m57.482644782s) [upload=6073]: Done starting upload Aug 7 22:22:43.895709036 (5h55m57.482765277s) [upload=6073]: Uploading segments... Aug 7 22:22:43.896017892 (5h55m57.483074133s) [upload=6073]: Got next segment. Inline: false Aug 7 22:22:43.896073596 (5h55m57.483129848s) [upload=6073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:43.896081962 (5h55m57.483138214s) [upload=6073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:23:43.881901614 (5h56m57.468957826s) [upload=6073]: Next segment err: context deadline exceeded Aug 7 22:23:43.882001040 (5h56m57.469057251s) [upload=6073]: Done uploading segments... Aug 7 22:23:43.882022099 (5h56m57.46907831s) [upload=6073]: Upload finished. err: context deadline exceeded Duration:* 1m0.001063366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 150.728776ms FAIL (2025-08-07 22:23:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 150.750417ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXX.X..................................XXXXXXXXXXXXXXXX.] duration: 219.172123ms FAIL (2025-08-07 22:23:30 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 219.191568ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX.............................XXXXXXXXXXXXXXXX] duration: 30.000778449s FAIL (2025-08-07 22:23:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ed Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:41.748643128 (5h50m55.33569938s) [upload=5998]: Starting upload Aug 7 22:17:41.748800843 (5h50m55.335857084s) [upload=5998]: Done starting upload Aug 7 22:17:41.748866916 (5h50m55.335923157s) [upload=5998]: Uploading segments... Aug 7 22:17:41.748948899 (5h50m55.33600514s) [upload=5998]: Got next segment. Inline: true Aug 7 22:17:41.748956914 (5h50m55.336013155s) [upload=5998]: Waiting for error group managing segments... Aug 7 22:17:41.749006376 (5h50m55.336062618s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:41.749012397 (5h50m55.336068639s) [upload=5998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:41.749016225 (5h50m55.336072466s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:11.736773915 (5h51m25.323830136s) [upload=5998]: Done uploading segments... Aug 7 22:18:11.736788051 (5h51m25.323844272s) [upload=5998]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:48.986206344 (5h52m2.573262556s) [upload=6015]: Starting upload Aug 7 22:18:48.986264964 (5h52m2.573321176s) [upload=6015]: Done starting upload Aug 7 22:18:48.986302574 (5h52m2.573358785s) [upload=6015]: Uploading segments... Aug 7 22:18:48.986327040 (5h52m2.573383251s) [upload=6015]: Got next segment. Inline: true Aug 7 22:18:48.986330306 (5h52m2.573386517s) [upload=6015]: Waiting for error group managing segments... Aug 7 22:18:48.986358459 (5h52m2.57341467s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:48.986361795 (5h52m2.573418006s) [upload=6015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:48.986364019 (5h52m2.573420231s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:18.972022741 (5h52m32.559078983s) [upload=6015]: Done uploading segments... Aug 7 22:19:18.972044513 (5h52m32.559100753s) [upload=6015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:33.193306109 (5h52m46.78036237s) [upload=6027]: Starting upload Aug 7 22:19:33.193429379 (5h52m46.78048563s) [upload=6027]: Done starting upload Aug 7 22:19:33.193516863 (5h52m46.780573103s) [upload=6027]: Uploading segments... Aug 7 22:19:33.193610827 (5h52m46.780667099s) [upload=6027]: Got next segment. Inline: true Aug 7 22:19:33.193618262 (5h52m46.780674513s) [upload=6027]: Waiting for error group managing segments... Aug 7 22:19:33.193668956 (5h52m46.780725196s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:33.193675307 (5h52m46.780731558s) [upload=6027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:33.193678734 (5h52m46.780734975s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:03.177682259 (5h53m16.76473848s) [upload=6027]: Done uploading segments... Aug 7 22:20:03.177699481 (5h53m16.764755702s) [upload=6027]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:33.558255550 (5h53m47.145311761s) [upload=6040]: Starting upload Aug 7 22:20:33.558305072 (5h53m47.145361293s) [upload=6040]: Done starting upload Aug 7 22:20:33.558352641 (5h53m47.145408852s) [upload=6040]: Uploading segments... Aug 7 22:20:33.558381975 (5h53m47.145438186s) [upload=6040]: Got next segment. Inline: true Aug 7 22:20:33.558385782 (5h53m47.145441993s) [upload=6040]: Waiting for error group managing segments... Aug 7 22:20:33.558417922 (5h53m47.145474134s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:33.558421789 (5h53m47.14547801s) [upload=6040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:33.558424405 (5h53m47.145480615s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:03.541933746 (5h54m17.128989987s) [upload=6040]: Done uploading segments... Aug 7 22:21:03.541952551 (5h54m17.129008792s) [upload=6040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:36.389854096 (5h54m49.976910308s) [upload=6055]: Starting upload Aug 7 22:21:36.389940908 (5h54m49.976997159s) [upload=6055]: Done starting upload Aug 7 22:21:36.390006921 (5h54m49.977063142s) [upload=6055]: Uploading segments... Aug 7 22:21:36.390051064 (5h54m49.977107285s) [upload=6055]: Got next segment. Inline: true Aug 7 22:21:36.390056905 (5h54m49.977113126s) [upload=6055]: Waiting for error group managing segments... Aug 7 22:21:36.390101748 (5h54m49.977157969s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:36.390107970 (5h54m49.977164191s) [upload=6055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:36.390112067 (5h54m49.977168288s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:06.377133119 (5h55m19.96418934s) [upload=6055]: Done uploading segments... Aug 7 22:22:06.377148799 (5h55m19.96420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00078931s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XX.........................................XXXXXXXXXXXXXXX] duration: 30.001277542s FAIL (2025-08-07 22:23:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001297699s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X.XX...............................XXXXXXXXXX] duration: 1m0.00152853s FAIL (2025-08-07 22:23:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:04.228585182 (5h50m17.815641403s) [upload=5990]: Starting upload Aug 7 22:17:04.228636998 (5h50m17.815693219s) [upload=5990]: Done starting upload Aug 7 22:17:04.228690769 (5h50m17.81574699s) [upload=5990]: Uploading segments... Aug 7 22:17:04.228721386 (5h50m17.815777607s) [upload=5990]: Got next segment. Inline: true Aug 7 22:17:04.228725082 (5h50m17.815781303s) [upload=5990]: Waiting for error group managing segments... Aug 7 22:17:04.228765939 (5h50m17.815822161s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:04.228772071 (5h50m17.815828292s) [upload=5990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:04.228775867 (5h50m17.815832089s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:04.215607888 (5h51m17.802664109s) [upload=5990]: Done uploading segments... Aug 7 22:18:04.215625421 (5h51m17.802681632s) [upload=5990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:04.292132983 (5h51m17.879189234s) [upload=6004]: Starting upload Aug 7 22:18:04.292228702 (5h51m17.879284943s) [upload=6004]: Done starting upload Aug 7 22:18:04.292318189 (5h51m17.879374439s) [upload=6004]: Uploading segments... Aug 7 22:18:04.292362020 (5h51m17.879418262s) [upload=6004]: Got next segment. Inline: true Aug 7 22:18:04.292367411 (5h51m17.879423662s) [upload=6004]: Waiting for error group managing segments... Aug 7 22:18:04.292410401 (5h51m17.879466651s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:04.292416432 (5h51m17.879472673s) [upload=6004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:04.292419948 (5h51m17.8794762s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:04.275395564 (5h52m17.862451785s) [upload=6004]: Done uploading segments... Aug 7 22:19:04.275413478 (5h52m17.862469689s) [upload=6004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001540833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXX...X.....X..............................XXXXXXXXXXXXXX] duration: 30.000955752s FAIL (2025-08-07 22:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000973084s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 230.164033ms FAIL (2025-08-07 22:23:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 230.183048ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXX.X..................................XXXXXXXXXXXXXXXX] duration: 30.001063915s FAIL (2025-08-07 22:23:05 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001082891s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX..................................XXX] duration: 30.000774377s FAIL (2025-08-07 22:23:03 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000790577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X........................................XXXXXXXXXXXXXX.] duration: 415.031983ms FAIL (2025-08-07 22:23:00 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== H3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218875761 (5h51m47.805931972s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219078590 (5h51m47.8061348s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219304070 (5h51m47.806360281s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.295781215 (5h51m47.882837427s) [upload=6008]: Next returned nil segment Aug 7 22:18:34.295793007 (5h51m47.882849228s) [upload=6008]: Waiting for error group managing segments... Aug 7 22:18:34.304725301 (5h51m47.891781522s) [noise=true , address=45.91.116.9:4261 , piece_id=3WFEHN2ORIVB3M5DRGZ4YU7POCTHAGZGOQC3PAOGQXSLH6Y26L2Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.305672389 (5h51m47.892728599s) [noise=true , address=77.90.31.16:4735 , piece_id=OJE2YSSCEUMDQYPIY3LAHXUGNRUINZH3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.306145892 (5h51m47.893202113s) [noise=true , address=45.91.116.6:4975 , piece_id=OIOLWW2NW7LLY5PCLQIACXX446OMDGK4SWK2DRO2AYKFYQJKNXOA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308384481 (5h51m47.895440692s) [noise=true , address=45.91.116.5:4297 , piece_id=5NFV2C77XBDXO6ROEZHIN3GJBL7T5ESI5PFDMF6YS5SG3IP4F7HQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308638956 (5h51m47.895695177s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.313407447 (5h51m47.900463668s) [noise=true , address=77.90.31.19:4843 , piece_id=C5QRMAC2OYAVQV2CQCNMKV74OLCZQI7637BBQNGZUSZJMUZ6OKMA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314255881 (5h51m47.901312102s) [noise=true , address=45.91.116.4:4938 , piece_id=2KCDC7EI5OLYGWZNGTEGHJFLAXFZU43QWPGZMYWGUUIBUVFK46DA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314483736 (5h51m47.901539957s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314993247 (5h51m47.902049468s) [noise=true , address=77.90.31.18:4805 , piece_id=BLRUHX7LZLOL5CIFUZK26T342Y7BWVMCCKTFBXIBJK3ZN7PPNFWA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.316405934 (5h51m47.903462145s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317679691 (5h51m47.904735903s) [noise=true , address=77.90.31.19:4842 , piece_id=RJUV6SSQFBCND6VC2U3CSJPYVHF5EAVAQSEMUBWS7YRDT6TZG3IA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317816056 (5h51m47.904872267s) [noise=true , address=77.90.31.10:4519 , piece_id=MT3UVFKYKHP55PC53ZE3TP63NYYDZKWPEFB4SB4AO5V4A7MPS5FQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327000981 (5h51m47.914057192s) [noise=true , address=104.36.82.16:4293 , piece_id=JF22AUONWX7BRRPMZBHD3ZKZ23S6A4VZ2PTOFRI2AH6A4F7VWIUA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327017582 (5h51m47.914073803s) [seg_pos={0 0}, upload=6008]: Segment reached optimal threshold of 13 pieces. Aug 7 22:18:34.327102280 (5h51m47.914158491s) [noise=true , address=77.90.31.13:4627 , piece_id=XPNONUA3EDJETDPLWMHVGDPB67S3AXRR4NB3HI4JUO3AKTTVUZAA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327167372 (5h51m47.914223593s) [noise=true , address=45.91.116.13:4154 , piece_id=YVSXAEDDGM7QG3DEECXTXL7D3VHWS53X2RSAM7LB5ZLCDKIV6R5A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327251158 (5h51m47.914307379s) [noise=true , address=104.36.82.15:4258 , piece_id=SUCLRXA647RFRWRWUTS66UIZ5AH37EJYPVUDSS6MMHA5ZPLSHE6Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: read tcp 5.161.89.252:46936->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:18:34.327269482 (5h51m47.914325693s) [seg_pos={0 0}, upload=6008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:18:34.327334693 (5h51m47.914390905s) [upload=6008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:18:34.327338841 (5h51m47.914395063s) [upload=6008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:25.781636440 (5h52m39.36869267s) [upload=6008]: Done uploading segments... Aug 7 22:19:25.781660244 (5h52m39.368716475s) [upload=6008]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:25.865723779 (5h52m39.45278003s) [upload=6025]: Starting upload Aug 7 22:19:25.865818526 (5h52m39.452874766s) [upload=6025]: Done starting upload Aug 7 22:19:25.865911980 (5h52m39.452968231s) [upload=6025]: Uploading segments... Aug 7 22:19:25.866235273 (5h52m39.453291524s) [upload=6025]: Got next segment. Inline: false Aug 7 22:19:25.866300124 (5h52m39.453356376s) [upload=6025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:25.866309232 (5h52m39.453365483s) [upload=6025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:55.843880801 (5h53m9.430937051s) [upload=6025]: Next segment err: context deadline exceeded Aug 7 22:19:55.843994953 (5h53m9.431051205s) [upload=6025]: Done uploading segments... Aug 7 22:19:55.844036191 (5h53m9.431092442s) [upload=6025]: Upload finished. err: context deadline exceeded Aug 7 22:20:00.803027956 (5h53m14.390084177s) [upload=6031]: Starting upload Aug 7 22:20:00.803087586 (5h53m14.390143807s) [upload=6031]: Done starting upload Aug 7 22:20:00.803139134 (5h53m14.390195344s) [upload=6031]: Uploading segments... Aug 7 22:20:00.803370255 (5h53m14.390426466s) [upload=6031]: Got next segment. Inline: false Aug 7 22:20:00.803406381 (5h53m14.390462602s) [upload=6031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:00.803410499 (5h53m14.390466711s) [upload=6031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:30.783569902 (5h53m44.370626154s) [upload=6031]: Next segment err: context deadline exceeded Aug 7 22:20:30.783672814 (5h53m44.370729066s) [upload=6031]: Done uploading segments... Aug 7 22:20:30.783702600 (5h53m44.370758842s) [upload=6031]: Upload finished. err: context deadline exceeded Aug 7 22:20:30.866635605 (5h53m44.453691847s) [upload=6038]: Starting upload Aug 7 22:20:30.866714704 (5h53m44.453770955s) [upload=6038]: Done starting upload Aug 7 22:20:30.866767612 (5h53m44.453823853s) [upload=6038]: Uploading segments... Aug 7 22:20:30.867141971 (5h53m44.454198223s) [upload=6038]: Got next segment. Inline: false Aug 7 22:20:30.867310425 (5h53m44.454366676s) [upload=6038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:30.867321436 (5h53m44.454377677s) [upload=6038]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:00.852060779 (5h54m14.43911699s) [upload=6038]: Next segment err: context deadline exceeded Aug 7 22:21:00.852158170 (5h54m14.439214391s) [upload=6038]: Done uploading segments... Aug 7 22:21:00.852219626 (5h54m14.439275847s) [upload=6038]: Upload finished. err: context deadline exceeded Aug 7 22:21:25.241993800 (5h54m38.829050021s) [upload=6052]: Starting upload Aug 7 22:21:25.242047039 (5h54m38.82910326s) [upload=6052]: Done starting upload Aug 7 22:21:25.242114276 (5h54m38.829170497s) [upload=6052]: Uploading segments... Aug 7 22:21:25.242294110 (5h54m38.829350331s) [upload=6052]: Got next segment. Inline: false Aug 7 22:21:25.242324568 (5h54m38.829380789s) [upload=6052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:25.242328595 (5h54m38.829384817s) [upload=6052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:55.229702839 (5h55m8.81675906s) [upload=6052]: Next segment err: context deadline exceeded Aug 7 22:21:55.229878175 (5h55m8.816934387s) [upload=6052]: Done uploading segments... Aug 7 22:21:55.229900377 (5h55m8.816956579s) [upload=6052]: Upload finished. err: context deadline exceeded Aug 7 22:22:20.437504693 (5h55m34.024560914s) [upload=6067]: Starting upload Aug 7 22:22:20.437554505 (5h55m34.024610716s) [upload=6067]: Done starting upload Aug 7 22:22:20.437642029 (5h55m34.02469825s) [upload=6067]: Uploading segments... Aug 7 22:22:20.437775748 (5h55m34.024831969s) [upload=6067]: Got next segment. Inline: false Aug 7 22:22:20.437814781 (5h55m34.024871002s) [upload=6067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:20.437818448 (5h55m34.024874659s) [upload=6067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:22:50.428010711 (5h56m4.015066932s) [upload=6067]: Next segment err: context deadline exceeded Aug 7 22:22:50.428069551 (5h56m4.015125782s) [upload=6067]: Done uploading segments... Aug 7 22:22:50.428091883 (5h56m4.015148104s) [upload=6067]: Upload finished. err: context deadline exceeded Duration: 415.060496ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 2m0.001113107s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:22:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001133064s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX..X........................................XXXXXXXXXXXXXX] duration: 30.002509856s FAIL (2025-08-07 22:22:50 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== H3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218875761 (5h51m47.805931972s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219078590 (5h51m47.8061348s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219304070 (5h51m47.806360281s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.295781215 (5h51m47.882837427s) [upload=6008]: Next returned nil segment Aug 7 22:18:34.295793007 (5h51m47.882849228s) [upload=6008]: Waiting for error group managing segments... Aug 7 22:18:34.304725301 (5h51m47.891781522s) [noise=true , address=45.91.116.9:4261 , piece_id=3WFEHN2ORIVB3M5DRGZ4YU7POCTHAGZGOQC3PAOGQXSLH6Y26L2Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.305672389 (5h51m47.892728599s) [noise=true , address=77.90.31.16:4735 , piece_id=OJE2YSSCEUMDQYPIY3LAHXUGNRUINZH3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.306145892 (5h51m47.893202113s) [noise=true , address=45.91.116.6:4975 , piece_id=OIOLWW2NW7LLY5PCLQIACXX446OMDGK4SWK2DRO2AYKFYQJKNXOA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308384481 (5h51m47.895440692s) [noise=true , address=45.91.116.5:4297 , piece_id=5NFV2C77XBDXO6ROEZHIN3GJBL7T5ESI5PFDMF6YS5SG3IP4F7HQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308638956 (5h51m47.895695177s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.313407447 (5h51m47.900463668s) [noise=true , address=77.90.31.19:4843 , piece_id=C5QRMAC2OYAVQV2CQCNMKV74OLCZQI7637BBQNGZUSZJMUZ6OKMA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314255881 (5h51m47.901312102s) [noise=true , address=45.91.116.4:4938 , piece_id=2KCDC7EI5OLYGWZNGTEGHJFLAXFZU43QWPGZMYWGUUIBUVFK46DA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314483736 (5h51m47.901539957s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314993247 (5h51m47.902049468s) [noise=true , address=77.90.31.18:4805 , piece_id=BLRUHX7LZLOL5CIFUZK26T342Y7BWVMCCKTFBXIBJK3ZN7PPNFWA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.316405934 (5h51m47.903462145s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317679691 (5h51m47.904735903s) [noise=true , address=77.90.31.19:4842 , piece_id=RJUV6SSQFBCND6VC2U3CSJPYVHF5EAVAQSEMUBWS7YRDT6TZG3IA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317816056 (5h51m47.904872267s) [noise=true , address=77.90.31.10:4519 , piece_id=MT3UVFKYKHP55PC53ZE3TP63NYYDZKWPEFB4SB4AO5V4A7MPS5FQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327000981 (5h51m47.914057192s) [noise=true , address=104.36.82.16:4293 , piece_id=JF22AUONWX7BRRPMZBHD3ZKZ23S6A4VZ2PTOFRI2AH6A4F7VWIUA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327017582 (5h51m47.914073803s) [seg_pos={0 0}, upload=6008]: Segment reached optimal threshold of 13 pieces. Aug 7 22:18:34.327102280 (5h51m47.914158491s) [noise=true , address=77.90.31.13:4627 , piece_id=XPNONUA3EDJETDPLWMHVGDPB67S3AXRR4NB3HI4JUO3AKTTVUZAA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327167372 (5h51m47.914223593s) [noise=true , address=45.91.116.13:4154 , piece_id=YVSXAEDDGM7QG3DEECXTXL7D3VHWS53X2RSAM7LB5ZLCDKIV6R5A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327251158 (5h51m47.914307379s) [noise=true , address=104.36.82.15:4258 , piece_id=SUCLRXA647RFRWRWUTS66UIZ5AH37EJYPVUDSS6MMHA5ZPLSHE6Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: read tcp 5.161.89.252:46936->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:18:34.327269482 (5h51m47.914325693s) [seg_pos={0 0}, upload=6008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:18:34.327334693 (5h51m47.914390905s) [upload=6008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:18:34.327338841 (5h51m47.914395063s) [upload=6008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:25.781636440 (5h52m39.36869267s) [upload=6008]: Done uploading segments... Aug 7 22:19:25.781660244 (5h52m39.368716475s) [upload=6008]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:25.865723779 (5h52m39.45278003s) [upload=6025]: Starting upload Aug 7 22:19:25.865818526 (5h52m39.452874766s) [upload=6025]: Done starting upload Aug 7 22:19:25.865911980 (5h52m39.452968231s) [upload=6025]: Uploading segments... Aug 7 22:19:25.866235273 (5h52m39.453291524s) [upload=6025]: Got next segment. Inline: false Aug 7 22:19:25.866300124 (5h52m39.453356376s) [upload=6025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:25.866309232 (5h52m39.453365483s) [upload=6025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:55.843880801 (5h53m9.430937051s) [upload=6025]: Next segment err: context deadline exceeded Aug 7 22:19:55.843994953 (5h53m9.431051205s) [upload=6025]: Done uploading segments... Aug 7 22:19:55.844036191 (5h53m9.431092442s) [upload=6025]: Upload finished. err: context deadline exceeded Aug 7 22:20:00.803027956 (5h53m14.390084177s) [upload=6031]: Starting upload Aug 7 22:20:00.803087586 (5h53m14.390143807s) [upload=6031]: Done starting upload Aug 7 22:20:00.803139134 (5h53m14.390195344s) [upload=6031]: Uploading segments... Aug 7 22:20:00.803370255 (5h53m14.390426466s) [upload=6031]: Got next segment. Inline: false Aug 7 22:20:00.803406381 (5h53m14.390462602s) [upload=6031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:00.803410499 (5h53m14.390466711s) [upload=6031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:30.783569902 (5h53m44.370626154s) [upload=6031]: Next segment err: context deadline exceeded Aug 7 22:20:30.783672814 (5h53m44.370729066s) [upload=6031]: Done uploading segments... Aug 7 22:20:30.783702600 (5h53m44.370758842s) [upload=6031]: Upload finished. err: context deadline exceeded Aug 7 22:20:30.866635605 (5h53m44.453691847s) [upload=6038]: Starting upload Aug 7 22:20:30.866714704 (5h53m44.453770955s) [upload=6038]: Done starting upload Aug 7 22:20:30.866767612 (5h53m44.453823853s) [upload=6038]: Uploading segments... Aug 7 22:20:30.867141971 (5h53m44.454198223s) [upload=6038]: Got next segment. Inline: false Aug 7 22:20:30.867310425 (5h53m44.454366676s) [upload=6038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:30.867321436 (5h53m44.454377677s) [upload=6038]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:00.852060779 (5h54m14.43911699s) [upload=6038]: Next segment err: context deadline exceeded Aug 7 22:21:00.852158170 (5h54m14.439214391s) [upload=6038]: Done uploading segments... Aug 7 22:21:00.852219626 (5h54m14.439275847s) [upload=6038]: Upload finished. err: context deadline exceeded Aug 7 22:21:25.241993800 (5h54m38.829050021s) [upload=6052]: Starting upload Aug 7 22:21:25.242047039 (5h54m38.82910326s) [upload=6052]: Done starting upload Aug 7 22:21:25.242114276 (5h54m38.829170497s) [upload=6052]: Uploading segments... Aug 7 22:21:25.242294110 (5h54m38.829350331s) [upload=6052]: Got next segment. Inline: false Aug 7 22:21:25.242324568 (5h54m38.829380789s) [upload=6052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:25.242328595 (5h54m38.829384817s) [upload=6052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:55.229702839 (5h55m8.81675906s) [upload=6052]: Next segment err: context deadline exceeded Aug 7 22:21:55.229878175 (5h55m8.816934387s) [upload=6052]: Done uploading segments... Aug 7 22:21:55.229900377 (5h55m8.816956579s) [upload=6052]: Upload finished. err: context deadline exceeded Aug 7 22:22:20.437504693 (5h55m34.024560914s) [upload=6067]: Starting upload Aug 7 22:22:20.437554505 (5h55m34.024610716s) [upload=6067]: Done starting upload Aug 7 22:22:20.437642029 (5h55m34.02469825s) [upload=6067]: Uploading segments... Aug 7 22:22:20.437775748 (5h55m34.024831969s) [upload=6067]: Got next segment. Inline: false Aug 7 22:22:20.437814781 (5h55m34.024871002s) [upload=6067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:20.437818448 (5h55m34.024874659s) [upload=6067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:22:50.428010711 (5h56m4.015066932s) [upload=6067]: Next segment err: context deadline exceeded Aug 7 22:22:50.428069551 (5h56m4.015125782s) [upload=6067]: Done uploading segments... Aug 7 22:22:50.428091883 (5h56m4.015148104s) [upload=6067]: Upload finished. err: context deadline exceeded Duration: 30.002520095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX....................................XXXXXXXXXXX] duration: 1m0.001279803s FAIL (2025-08-07 22:22:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001295092s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [X....................................................X......] duration: 113.635785ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:22:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 113.658598ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXX.............................XXXXXXXXXXXXXXX] duration: 30.000595476s FAIL (2025-08-07 22:22:36 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:41.748643128 (5h50m55.33569938s) [upload=5998]: Starting upload Aug 7 22:17:41.748800843 (5h50m55.335857084s) [upload=5998]: Done starting upload Aug 7 22:17:41.748866916 (5h50m55.335923157s) [upload=5998]: Uploading segments... Aug 7 22:17:41.748948899 (5h50m55.33600514s) [upload=5998]: Got next segment. Inline: true Aug 7 22:17:41.748956914 (5h50m55.336013155s) [upload=5998]: Waiting for error group managing segments... Aug 7 22:17:41.749006376 (5h50m55.336062618s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:41.749012397 (5h50m55.336068639s) [upload=5998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:41.749016225 (5h50m55.336072466s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:11.736773915 (5h51m25.323830136s) [upload=5998]: Done uploading segments... Aug 7 22:18:11.736788051 (5h51m25.323844272s) [upload=5998]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:48.986206344 (5h52m2.573262556s) [upload=6015]: Starting upload Aug 7 22:18:48.986264964 (5h52m2.573321176s) [upload=6015]: Done starting upload Aug 7 22:18:48.986302574 (5h52m2.573358785s) [upload=6015]: Uploading segments... Aug 7 22:18:48.986327040 (5h52m2.573383251s) [upload=6015]: Got next segment. Inline: true Aug 7 22:18:48.986330306 (5h52m2.573386517s) [upload=6015]: Waiting for error group managing segments... Aug 7 22:18:48.986358459 (5h52m2.57341467s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:48.986361795 (5h52m2.573418006s) [upload=6015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:48.986364019 (5h52m2.573420231s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:18.972022741 (5h52m32.559078983s) [upload=6015]: Done uploading segments... Aug 7 22:19:18.972044513 (5h52m32.559100753s) [upload=6015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:33.193306109 (5h52m46.78036237s) [upload=6027]: Starting upload Aug 7 22:19:33.193429379 (5h52m46.78048563s) [upload=6027]: Done starting upload Aug 7 22:19:33.193516863 (5h52m46.780573103s) [upload=6027]: Uploading segments... Aug 7 22:19:33.193610827 (5h52m46.780667099s) [upload=6027]: Got next segment. Inline: true Aug 7 22:19:33.193618262 (5h52m46.780674513s) [upload=6027]: Waiting for error group managing segments... Aug 7 22:19:33.193668956 (5h52m46.780725196s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:33.193675307 (5h52m46.780731558s) [upload=6027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:33.193678734 (5h52m46.780734975s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:03.177682259 (5h53m16.76473848s) [upload=6027]: Done uploading segments... Aug 7 22:20:03.177699481 (5h53m16.764755702s) [upload=6027]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:33.558255550 (5h53m47.145311761s) [upload=6040]: Starting upload Aug 7 22:20:33.558305072 (5h53m47.145361293s) [upload=6040]: Done starting upload Aug 7 22:20:33.558352641 (5h53m47.145408852s) [upload=6040]: Uploading segments... Aug 7 22:20:33.558381975 (5h53m47.145438186s) [upload=6040]: Got next segment. Inline: true Aug 7 22:20:33.558385782 (5h53m47.145441993s) [upload=6040]: Waiting for error group managing segments... Aug 7 22:20:33.558417922 (5h53m47.145474134s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:33.558421789 (5h53m47.14547801s) [upload=6040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:33.558424405 (5h53m47.145480615s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:03.541933746 (5h54m17.128989987s) [upload=6040]: Done uploading segments... Aug 7 22:21:03.541952551 (5h54m17.129008792s) [upload=6040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:36.389854096 (5h54m49.976910308s) [upload=6055]: Starting upload Aug 7 22:21:36.389940908 (5h54m49.976997159s) [upload=6055]: Done starting upload Aug 7 22:21:36.390006921 (5h54m49.977063142s) [upload=6055]: Uploading segments... Aug 7 22:21:36.390051064 (5h54m49.977107285s) [upload=6055]: Got next segment. Inline: true Aug 7 22:21:36.390056905 (5h54m49.977113126s) [upload=6055]: Waiting for error group managing segments... Aug 7 22:21:36.390101748 (5h54m49.977157969s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:36.390107970 (5h54m49.977164191s) [upload=6055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:36.390112067 (5h54m49.977168288s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:06.377133119 (5h55m19.96418934s) [upload=6055]: Done uploading segments... Aug 7 22:22:06.377148799 (5h55m19.96420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00060978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX......X.............................XXXXXXXXXXX] duration: 1m0.001235436s FAIL (2025-08-07 22:22:31 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Aug 7 22:16:22.269422973 (5h49m35.856479254s) [upload=5981]: Starting upload Aug 7 22:16:22.269535553 (5h49m35.856591794s) [upload=5981]: Done starting upload Aug 7 22:16:22.269676246 (5h49m35.856732487s) [upload=5981]: Uploading segments... Aug 7 22:16:22.269948395 (5h49m35.857004635s) [upload=5981]: Got next segment. Inline: false Aug 7 22:16:22.269993167 (5h49m35.857049419s) [upload=5981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:22.269999790 (5h49m35.857056041s) [upload=5981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:22.257287873 (5h50m35.844344113s) [upload=5981]: Next segment err: context deadline exceeded Aug 7 22:17:22.257515257 (5h50m35.844571508s) [upload=5981]: Done uploading segments... Aug 7 22:17:22.257574649 (5h50m35.844630889s) [upload=5981]: Upload finished. err: context deadline exceeded Aug 7 22:17:22.352562567 (5h50m35.939618808s) [upload=5994]: Starting upload Aug 7 22:17:22.352642467 (5h50m35.939698707s) [upload=5994]: Done starting upload Aug 7 22:17:22.352739408 (5h50m35.939795648s) [upload=5994]: Uploading segments... Aug 7 22:17:22.353012627 (5h50m35.940068868s) [upload=5994]: Got next segment. Inline: false Aug 7 22:17:22.353063573 (5h50m35.940119824s) [upload=5994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:22.353069433 (5h50m35.940125675s) [upload=5994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:22.334060789 (5h51m35.92111701s) [upload=5994]: Next segment err: context deadline exceeded Aug 7 22:18:22.334109780 (5h51m35.921165992s) [upload=5994]: Done uploading segments... Aug 7 22:18:22.334124208 (5h51m35.921180419s) [upload=5994]: Upload finished. err: context deadline exceeded Aug 7 22:18:22.417801973 (5h51m36.004858194s) [upload=6010]: Starting upload Aug 7 22:18:22.417851666 (5h51m36.004907878s) [upload=6010]: Done starting upload Aug 7 22:18:22.417922909 (5h51m36.00497913s) [upload=6010]: Uploading segments... Aug 7 22:18:22.418063011 (5h51m36.005119222s) [upload=6010]: Got next segment. Inline: false Aug 7 22:18:22.418083209 (5h51m36.00513943s) [upload=6010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:22.418088378 (5h51m36.005144599s) [upload=6010]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:22.407229763 (5h52m35.994286003s) [upload=6010]: Next segment err: context deadline exceeded Aug 7 22:19:22.407403607 (5h52m35.994459847s) [upload=6010]: Done uploading segments... Aug 7 22:19:22.407430978 (5h52m35.994487219s) [upload=6010]: Upload finished. err: context deadline exceeded Aug 7 22:19:22.483258289 (5h52m36.07031451s) [upload=6023]: Starting upload Aug 7 22:19:22.483313302 (5h52m36.070369514s) [upload=6023]: Done starting upload Aug 7 22:19:22.483363717 (5h52m36.070419968s) [upload=6023]: Uploading segments... Aug 7 22:19:22.483555084 (5h52m36.070611295s) [upload=6023]: Got next segment. Inline: false Aug 7 22:19:22.483607642 (5h52m36.070663852s) [upload=6023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:22.483613111 (5h52m36.070669332s) [upload=6023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:22.470951760 (5h53m36.058008001s) [upload=6023]: Next segment err: context deadline exceeded Aug 7 22:20:22.471159418 (5h53m36.058215659s) [upload=6023]: Done uploading segments... Aug 7 22:20:22.471229579 (5h53m36.058285831s) [upload=6023]: Upload finished. err: context deadline exceeded Aug 7 22:20:31.584716452 (5h53m45.171772692s) [upload=6039]: Starting upload Aug 7 22:20:31.584819494 (5h53m45.171875734s) [upload=6039]: Done starting upload Aug 7 22:20:31.584946481 (5h53m45.172002722s) [upload=6039]: Uploading segments... Aug 7 22:20:31.585228928 (5h53m45.172285168s) [upload=6039]: Got next segment. Inline: false Aug 7 22:20:31.585296344 (5h53m45.172352585s) [upload=6039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:31.585305191 (5h53m45.172361432s) [upload=6039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:31.564329759 (5h54m45.15138597s) [upload=6039]: Next segment err: context deadline exceeded Aug 7 22:21:31.564470662 (5h54m45.151526883s) [upload=6039]: Done uploading segments... Aug 7 22:21:31.564486933 (5h54m45.151543144s) [upload=6039]: Upload finished. err: context deadline exceeded Aug 7 22:21:31.630894751 (5h54m45.217950972s) [upload=6054]: Starting upload Aug 7 22:21:31.630942149 (5h54m45.21799837s) [upload=6054]: Done starting upload Aug 7 22:21:31.631165055 (5h54m45.218221276s) [upload=6054]: Uploading segments... Aug 7 22:21:31.631364788 (5h54m45.218421009s) [upload=6054]: Got next segment. Inline: false Aug 7 22:21:31.631413229 (5h54m45.21846945s) [upload=6054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:31.631419942 (5h54m45.218476182s) [upload=6054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:22:31.618721098 (5h55m45.205777349s) [upload=6054]: Next segment err: context deadline exceeded Aug 7 22:22:31.618829049 (5h55m45.20588529s) [upload=6054]: Done uploading segments... Aug 7 22:22:31.618858314 (5h55m45.205914555s) [upload=6054]: Upload finished. err: context deadline exceeded Duration: 1m0.001256154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XX.........................................XXXXXXXXXXXXXX] duration: 30.00169905s FAIL (2025-08-07 22:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00173074s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXX...X.....X..............................XXXXXXXXXXXXX] duration: 30.000579349s FAIL (2025-08-07 22:22:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000595289s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX..................................XX] duration: 30.001284231s FAIL (2025-08-07 22:22:15 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001296414s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X.XX...............................XXXXXXXXX] duration: 1m0.001829024s FAIL (2025-08-07 22:22:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:04.228585182 (5h50m17.815641403s) [upload=5990]: Starting upload Aug 7 22:17:04.228636998 (5h50m17.815693219s) [upload=5990]: Done starting upload Aug 7 22:17:04.228690769 (5h50m17.81574699s) [upload=5990]: Uploading segments... Aug 7 22:17:04.228721386 (5h50m17.815777607s) [upload=5990]: Got next segment. Inline: true Aug 7 22:17:04.228725082 (5h50m17.815781303s) [upload=5990]: Waiting for error group managing segments... Aug 7 22:17:04.228765939 (5h50m17.815822161s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:04.228772071 (5h50m17.815828292s) [upload=5990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:04.228775867 (5h50m17.815832089s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:04.215607888 (5h51m17.802664109s) [upload=5990]: Done uploading segments... Aug 7 22:18:04.215625421 (5h51m17.802681632s) [upload=5990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:04.292132983 (5h51m17.879189234s) [upload=6004]: Starting upload Aug 7 22:18:04.292228702 (5h51m17.879284943s) [upload=6004]: Done starting upload Aug 7 22:18:04.292318189 (5h51m17.879374439s) [upload=6004]: Uploading segments... Aug 7 22:18:04.292362020 (5h51m17.879418262s) [upload=6004]: Got next segment. Inline: true Aug 7 22:18:04.292367411 (5h51m17.879423662s) [upload=6004]: Waiting for error group managing segments... Aug 7 22:18:04.292410401 (5h51m17.879466651s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:04.292416432 (5h51m17.879472673s) [upload=6004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:04.292419948 (5h51m17.8794762s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:04.275395564 (5h52m17.862451785s) [upload=6004]: Done uploading segments... Aug 7 22:19:04.275413478 (5h52m17.862469689s) [upload=6004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001840656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXX.............................XXXXXXXXXXXXXX] duration: 30.000626479s FAIL (2025-08-07 22:22:06 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== eadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:41.748643128 (5h50m55.33569938s) [upload=5998]: Starting upload Aug 7 22:17:41.748800843 (5h50m55.335857084s) [upload=5998]: Done starting upload Aug 7 22:17:41.748866916 (5h50m55.335923157s) [upload=5998]: Uploading segments... Aug 7 22:17:41.748948899 (5h50m55.33600514s) [upload=5998]: Got next segment. Inline: true Aug 7 22:17:41.748956914 (5h50m55.336013155s) [upload=5998]: Waiting for error group managing segments... Aug 7 22:17:41.749006376 (5h50m55.336062618s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:41.749012397 (5h50m55.336068639s) [upload=5998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:41.749016225 (5h50m55.336072466s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:11.736773915 (5h51m25.323830136s) [upload=5998]: Done uploading segments... Aug 7 22:18:11.736788051 (5h51m25.323844272s) [upload=5998]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:48.986206344 (5h52m2.573262556s) [upload=6015]: Starting upload Aug 7 22:18:48.986264964 (5h52m2.573321176s) [upload=6015]: Done starting upload Aug 7 22:18:48.986302574 (5h52m2.573358785s) [upload=6015]: Uploading segments... Aug 7 22:18:48.986327040 (5h52m2.573383251s) [upload=6015]: Got next segment. Inline: true Aug 7 22:18:48.986330306 (5h52m2.573386517s) [upload=6015]: Waiting for error group managing segments... Aug 7 22:18:48.986358459 (5h52m2.57341467s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:48.986361795 (5h52m2.573418006s) [upload=6015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:48.986364019 (5h52m2.573420231s) [upload=6015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:18.972022741 (5h52m32.559078983s) [upload=6015]: Done uploading segments... Aug 7 22:19:18.972044513 (5h52m32.559100753s) [upload=6015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:33.193306109 (5h52m46.78036237s) [upload=6027]: Starting upload Aug 7 22:19:33.193429379 (5h52m46.78048563s) [upload=6027]: Done starting upload Aug 7 22:19:33.193516863 (5h52m46.780573103s) [upload=6027]: Uploading segments... Aug 7 22:19:33.193610827 (5h52m46.780667099s) [upload=6027]: Got next segment. Inline: true Aug 7 22:19:33.193618262 (5h52m46.780674513s) [upload=6027]: Waiting for error group managing segments... Aug 7 22:19:33.193668956 (5h52m46.780725196s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:33.193675307 (5h52m46.780731558s) [upload=6027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:33.193678734 (5h52m46.780734975s) [upload=6027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:03.177682259 (5h53m16.76473848s) [upload=6027]: Done uploading segments... Aug 7 22:20:03.177699481 (5h53m16.764755702s) [upload=6027]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:33.558255550 (5h53m47.145311761s) [upload=6040]: Starting upload Aug 7 22:20:33.558305072 (5h53m47.145361293s) [upload=6040]: Done starting upload Aug 7 22:20:33.558352641 (5h53m47.145408852s) [upload=6040]: Uploading segments... Aug 7 22:20:33.558381975 (5h53m47.145438186s) [upload=6040]: Got next segment. Inline: true Aug 7 22:20:33.558385782 (5h53m47.145441993s) [upload=6040]: Waiting for error group managing segments... Aug 7 22:20:33.558417922 (5h53m47.145474134s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:33.558421789 (5h53m47.14547801s) [upload=6040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:33.558424405 (5h53m47.145480615s) [upload=6040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:03.541933746 (5h54m17.128989987s) [upload=6040]: Done uploading segments... Aug 7 22:21:03.541952551 (5h54m17.129008792s) [upload=6040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:36.389854096 (5h54m49.976910308s) [upload=6055]: Starting upload Aug 7 22:21:36.389940908 (5h54m49.976997159s) [upload=6055]: Done starting upload Aug 7 22:21:36.390006921 (5h54m49.977063142s) [upload=6055]: Uploading segments... Aug 7 22:21:36.390051064 (5h54m49.977107285s) [upload=6055]: Got next segment. Inline: true Aug 7 22:21:36.390056905 (5h54m49.977113126s) [upload=6055]: Waiting for error group managing segments... Aug 7 22:21:36.390101748 (5h54m49.977157969s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:36.390107970 (5h54m49.977164191s) [upload=6055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:36.390112067 (5h54m49.977168288s) [upload=6055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:06.377133119 (5h55m19.96418934s) [upload=6055]: Done uploading segments... Aug 7 22:22:06.377148799 (5h55m19.96420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000647087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXX.X..................................XXXXXXXXXXXXXXX] duration: 30.00067487s FAIL (2025-08-07 22:21:59 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000699746s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 127.692724ms FAIL (2025-08-07 22:21:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 127.714345ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.X....................................................X.....] duration: 108.107327ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:21:56 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 108.158652ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX..X........................................XXXXXXXXXXXXX] duration: 30.001132594s FAIL (2025-08-07 22:21:55 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 16.9:4261 , piece_id=3WFEHN2ORIVB3M5DRGZ4YU7POCTHAGZGOQC3PAOGQXSLH6Y26L2Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218444917 (5h51m47.805501138s) [noise=true , address=77.90.31.19:4843 , piece_id=C5QRMAC2OYAVQV2CQCNMKV74OLCZQI7637BBQNGZUSZJMUZ6OKMA, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218520938 (5h51m47.805577159s) [noise=true , address=77.90.31.19:4842 , piece_id=RJUV6SSQFBCND6VC2U3CSJPYVHF5EAVAQSEMUBWS7YRDT6TZG3IA, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218668324 (5h51m47.805724535s) [noise=true , address=77.90.31.13:4627 , piece_id=XPNONUA3EDJETDPLWMHVGDPB67S3AXRR4NB3HI4JUO3AKTTVUZAA, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218800720 (5h51m47.805856941s) [noise=true , address=77.90.31.16:4735 , piece_id=OJE2YSSCEUMDQYPIY3LAHXUGNRUINZH3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.218875761 (5h51m47.805931972s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219078590 (5h51m47.8061348s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.219304070 (5h51m47.806360281s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Uploading piece... Aug 7 22:18:34.295781215 (5h51m47.882837427s) [upload=6008]: Next returned nil segment Aug 7 22:18:34.295793007 (5h51m47.882849228s) [upload=6008]: Waiting for error group managing segments... Aug 7 22:18:34.304725301 (5h51m47.891781522s) [noise=true , address=45.91.116.9:4261 , piece_id=3WFEHN2ORIVB3M5DRGZ4YU7POCTHAGZGOQC3PAOGQXSLH6Y26L2Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.305672389 (5h51m47.892728599s) [noise=true , address=77.90.31.16:4735 , piece_id=OJE2YSSCEUMDQYPIY3LAHXUGNRUINZH3TJUMVNRZNPJO6NZPJS4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.306145892 (5h51m47.893202113s) [noise=true , address=45.91.116.6:4975 , piece_id=OIOLWW2NW7LLY5PCLQIACXX446OMDGK4SWK2DRO2AYKFYQJKNXOA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308384481 (5h51m47.895440692s) [noise=true , address=45.91.116.5:4297 , piece_id=5NFV2C77XBDXO6ROEZHIN3GJBL7T5ESI5PFDMF6YS5SG3IP4F7HQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.308638956 (5h51m47.895695177s) [noise=true , address=77.90.31.15:4697 , piece_id=Y5RGW3ORMDM6BLINPYXS4QI436BU5TAJWIY6IZNOXLZV4HBD5I2A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.313407447 (5h51m47.900463668s) [noise=true , address=77.90.31.19:4843 , piece_id=C5QRMAC2OYAVQV2CQCNMKV74OLCZQI7637BBQNGZUSZJMUZ6OKMA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314255881 (5h51m47.901312102s) [noise=true , address=45.91.116.4:4938 , piece_id=2KCDC7EI5OLYGWZNGTEGHJFLAXFZU43QWPGZMYWGUUIBUVFK46DA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314483736 (5h51m47.901539957s) [noise=true , address=77.90.31.17:4771 , piece_id=AU43BITJLRQM6LHMDTLZPISMDXDPJJQZL7H525K6EKEEKHV7Y3MQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.314993247 (5h51m47.902049468s) [noise=true , address=77.90.31.18:4805 , piece_id=BLRUHX7LZLOL5CIFUZK26T342Y7BWVMCCKTFBXIBJK3ZN7PPNFWA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.316405934 (5h51m47.903462145s) [noise=true , address=45.91.116.11:4118 , piece_id=PE4PK4JTPIUJ5UCBAZH73RKARSHWQLN26P34HMFLZSTDU54L6R4Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317679691 (5h51m47.904735903s) [noise=true , address=77.90.31.19:4842 , piece_id=RJUV6SSQFBCND6VC2U3CSJPYVHF5EAVAQSEMUBWS7YRDT6TZG3IA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.317816056 (5h51m47.904872267s) [noise=true , address=77.90.31.10:4519 , piece_id=MT3UVFKYKHP55PC53ZE3TP63NYYDZKWPEFB4SB4AO5V4A7MPS5FQ, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327000981 (5h51m47.914057192s) [noise=true , address=104.36.82.16:4293 , piece_id=JF22AUONWX7BRRPMZBHD3ZKZ23S6A4VZ2PTOFRI2AH6A4F7VWIUA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: <nil> Aug 7 22:18:34.327017582 (5h51m47.914073803s) [seg_pos={0 0}, upload=6008]: Segment reached optimal threshold of 13 pieces. Aug 7 22:18:34.327102280 (5h51m47.914158491s) [noise=true , address=77.90.31.13:4627 , piece_id=XPNONUA3EDJETDPLWMHVGDPB67S3AXRR4NB3HI4JUO3AKTTVUZAA, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327167372 (5h51m47.914223593s) [noise=true , address=45.91.116.13:4154 , piece_id=YVSXAEDDGM7QG3DEECXTXL7D3VHWS53X2RSAM7LB5ZLCDKIV6R5A, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 22:18:34.327251158 (5h51m47.914307379s) [noise=true , address=104.36.82.15:4258 , piece_id=SUCLRXA647RFRWRWUTS66UIZ5AH37EJYPVUDSS6MMHA5ZPLSHE6Q, seg_pos={0 0}, upload=6008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: read tcp 5.161.89.252:46936->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:18:34.327269482 (5h51m47.914325693s) [seg_pos={0 0}, upload=6008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:18:34.327334693 (5h51m47.914390905s) [upload=6008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:18:34.327338841 (5h51m47.914395063s) [upload=6008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:25.781636440 (5h52m39.36869267s) [upload=6008]: Done uploading segments... Aug 7 22:19:25.781660244 (5h52m39.368716475s) [upload=6008]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:25.865723779 (5h52m39.45278003s) [upload=6025]: Starting upload Aug 7 22:19:25.865818526 (5h52m39.452874766s) [upload=6025]: Done starting upload Aug 7 22:19:25.865911980 (5h52m39.452968231s) [upload=6025]: Uploading segments... Aug 7 22:19:25.866235273 (5h52m39.453291524s) [upload=6025]: Got next segment. Inline: false Aug 7 22:19:25.866300124 (5h52m39.453356376s) [upload=6025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:25.866309232 (5h52m39.453365483s) [upload=6025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:55.843880801 (5h53m9.430937051s) [upload=6025]: Next segment err: context deadline exceeded Aug 7 22:19:55.843994953 (5h53m9.431051205s) [upload=6025]: Done uploading segments... Aug 7 22:19:55.844036191 (5h53m9.431092442s) [upload=6025]: Upload finished. err: context deadline exceeded Aug 7 22:20:00.803027956 (5h53m14.390084177s) [upload=6031]: Starting upload Aug 7 22:20:00.803087586 (5h53m14.390143807s) [upload=6031]: Done starting upload Aug 7 22:20:00.803139134 (5h53m14.390195344s) [upload=6031]: Uploading segments... Aug 7 22:20:00.803370255 (5h53m14.390426466s) [upload=6031]: Got next segment. Inline: false Aug 7 22:20:00.803406381 (5h53m14.390462602s) [upload=6031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:00.803410499 (5h53m14.390466711s) [upload=6031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:30.783569902 (5h53m44.370626154s) [upload=6031]: Next segment err: context deadline exceeded Aug 7 22:20:30.783672814 (5h53m44.370729066s) [upload=6031]: Done uploading segments... Aug 7 22:20:30.783702600 (5h53m44.370758842s) [upload=6031]: Upload finished. err: context deadline exceeded Aug 7 22:20:30.866635605 (5h53m44.453691847s) [upload=6038]: Starting upload Aug 7 22:20:30.866714704 (5h53m44.453770955s) [upload=6038]: Done starting upload Aug 7 22:20:30.866767612 (5h53m44.453823853s) [upload=6038]: Uploading segments... Aug 7 22:20:30.867141971 (5h53m44.454198223s) [upload=6038]: Got next segment. Inline: false Aug 7 22:20:30.867310425 (5h53m44.454366676s) [upload=6038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:30.867321436 (5h53m44.454377677s) [upload=6038]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:00.852060779 (5h54m14.43911699s) [upload=6038]: Next segment err: context deadline exceeded Aug 7 22:21:00.852158170 (5h54m14.439214391s) [upload=6038]: Done uploading segments... Aug 7 22:21:00.852219626 (5h54m14.439275847s) [upload=6038]: Upload finished. err: context deadline exceeded Aug 7 22:21:25.241993800 (5h54m38.829050021s) [upload=6052]: Starting upload Aug 7 22:21:25.242047039 (5h54m38.82910326s) [upload=6052]: Done starting upload Aug 7 22:21:25.242114276 (5h54m38.829170497s) [upload=6052]: Uploading segments... Aug 7 22:21:25.242294110 (5h54m38.829350331s) [upload=6052]: Got next segment. Inline: false Aug 7 22:21:25.242324568 (5h54m38.829380789s) [upload=6052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:25.242328595 (5h54m38.829384817s) [upload=6052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:55.229702839 (5h55m8.81675906s) [upload=6052]: Next segment err: context deadline exceeded Aug 7 22:21:55.229878175 (5h55m8.816934387s) [upload=6052]: Done uploading segments... Aug 7 22:21:55.229900377 (5h55m8.816956579s) [upload=6052]: Upload finished. err: context deadline exceeded Duration: 30.001149716s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.XX.........................................XXXXXXXXXXXXX] duration: 30.000399399s FAIL (2025-08-07 22:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000409797s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX....................................XXXXXXXXXX] duration: 1m0.001281101s FAIL (2025-08-07 22:21:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001293555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX..................................X] duration: 10m48.458893517s FAIL (2025-08-07 22:21:45 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 10m48.45890579s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX......X.............................XXXXXXXXXX] duration: 1m0.001553562s FAIL (2025-08-07 22:21:31 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== rk connection; piecestore: piecestore close: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection Aug 7 22:10:00.596771750 (5h43m14.183827971s) [noise=true , address=104.36.82.37:4685 , piece_id=4Q4WHVJWI4VXAEZEPTS355RHEBI37ME5S3MDUI5KM45LVDZ3GZ3A, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM): protocol: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection Aug 7 22:10:00.596795714 (5h43m14.183851935s) [seg_pos={0 0}, upload=5882]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Aug 7 22:16:22.269422973 (5h49m35.856479254s) [upload=5981]: Starting upload Aug 7 22:16:22.269535553 (5h49m35.856591794s) [upload=5981]: Done starting upload Aug 7 22:16:22.269676246 (5h49m35.856732487s) [upload=5981]: Uploading segments... Aug 7 22:16:22.269948395 (5h49m35.857004635s) [upload=5981]: Got next segment. Inline: false Aug 7 22:16:22.269993167 (5h49m35.857049419s) [upload=5981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:22.269999790 (5h49m35.857056041s) [upload=5981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:22.257287873 (5h50m35.844344113s) [upload=5981]: Next segment err: context deadline exceeded Aug 7 22:17:22.257515257 (5h50m35.844571508s) [upload=5981]: Done uploading segments... Aug 7 22:17:22.257574649 (5h50m35.844630889s) [upload=5981]: Upload finished. err: context deadline exceeded Aug 7 22:17:22.352562567 (5h50m35.939618808s) [upload=5994]: Starting upload Aug 7 22:17:22.352642467 (5h50m35.939698707s) [upload=5994]: Done starting upload Aug 7 22:17:22.352739408 (5h50m35.939795648s) [upload=5994]: Uploading segments... Aug 7 22:17:22.353012627 (5h50m35.940068868s) [upload=5994]: Got next segment. Inline: false Aug 7 22:17:22.353063573 (5h50m35.940119824s) [upload=5994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:22.353069433 (5h50m35.940125675s) [upload=5994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:22.334060789 (5h51m35.92111701s) [upload=5994]: Next segment err: context deadline exceeded Aug 7 22:18:22.334109780 (5h51m35.921165992s) [upload=5994]: Done uploading segments... Aug 7 22:18:22.334124208 (5h51m35.921180419s) [upload=5994]: Upload finished. err: context deadline exceeded Aug 7 22:18:22.417801973 (5h51m36.004858194s) [upload=6010]: Starting upload Aug 7 22:18:22.417851666 (5h51m36.004907878s) [upload=6010]: Done starting upload Aug 7 22:18:22.417922909 (5h51m36.00497913s) [upload=6010]: Uploading segments... Aug 7 22:18:22.418063011 (5h51m36.005119222s) [upload=6010]: Got next segment. Inline: false Aug 7 22:18:22.418083209 (5h51m36.00513943s) [upload=6010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:22.418088378 (5h51m36.005144599s) [upload=6010]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:22.407229763 (5h52m35.994286003s) [upload=6010]: Next segment err: context deadline exceeded Aug 7 22:19:22.407403607 (5h52m35.994459847s) [upload=6010]: Done uploading segments... Aug 7 22:19:22.407430978 (5h52m35.994487219s) [upload=6010]: Upload finished. err: context deadline exceeded Aug 7 22:19:22.483258289 (5h52m36.07031451s) [upload=6023]: Starting upload Aug 7 22:19:22.483313302 (5h52m36.070369514s) [upload=6023]: Done starting upload Aug 7 22:19:22.483363717 (5h52m36.070419968s) [upload=6023]: Uploading segments... Aug 7 22:19:22.483555084 (5h52m36.070611295s) [upload=6023]: Got next segment. Inline: false Aug 7 22:19:22.483607642 (5h52m36.070663852s) [upload=6023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:22.483613111 (5h52m36.070669332s) [upload=6023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:22.470951760 (5h53m36.058008001s) [upload=6023]: Next segment err: context deadline exceeded Aug 7 22:20:22.471159418 (5h53m36.058215659s) [upload=6023]: Done uploading segments... Aug 7 22:20:22.471229579 (5h53m36.058285831s) [upload=6023]: Upload finished. err: context deadline exceeded Aug 7 22:20:31.584716452 (5h53m45.171772692s) [upload=6039]: Starting upload Aug 7 22:20:31.584819494 (5h53m45.171875734s) [upload=6039]: Done starting upload Aug 7 22:20:31.584946481 (5h53m45.172002722s) [upload=6039]: Uploading segments... Aug 7 22:20:31.585228928 (5h53m45.172285168s) [upload=6039]: Got next segment. Inline: false Aug 7 22:20:31.585296344 (5h53m45.172352585s) [upload=6039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:31.585305191 (5h53m45.172361432s) [upload=6039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:31.564329759 (5h54m45.15138597s) [upload=6039]: Next segment err: context deadline exceeded Aug 7 22:21:31.564470662 (5h54m45.151526883s) [upload=6039]: Done uploading segments... Aug 7 22:21:31.564486933 (5h54m45.151543144s) [upload=6039]: Upload finished. err: context deadline exceeded Duration: 1m0.00156881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.XX.........................................XXXXXXXXXXXX] duration: 30.001581399s FAIL (2025-08-07 22:21:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001605886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 162.799995ms FAIL (2025-08-07 22:21:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 162.820714ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X.XX...............................XXXXXXXX] duration: 1m0.001031387s FAIL (2025-08-07 22:21:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3055664s) [upload=5887]: Upload finished. err: <nil> Aug 7 22:11:00.429392635 (5h44m14.016448856s) [upload=5899]: Starting upload Aug 7 22:11:00.429436647 (5h44m14.016492868s) [upload=5899]: Done starting upload Aug 7 22:11:00.429474658 (5h44m14.016530879s) [upload=5899]: Uploading segments... Aug 7 22:11:00.429501167 (5h44m14.016557378s) [upload=5899]: Got next segment. Inline: true Aug 7 22:11:00.429504493 (5h44m14.016560714s) [upload=5899]: Waiting for error group managing segments... Aug 7 22:11:00.429533688 (5h44m14.016589909s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:00.429537505 (5h44m14.016593717s) [upload=5899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:00.429540060 (5h44m14.016596281s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:00.493866886 (5h44m14.080923097s) [upload=5899]: Done uploading segments... Aug 7 22:11:00.493879859 (5h44m14.08093608s) [upload=5899]: Upload finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:04.228585182 (5h50m17.815641403s) [upload=5990]: Starting upload Aug 7 22:17:04.228636998 (5h50m17.815693219s) [upload=5990]: Done starting upload Aug 7 22:17:04.228690769 (5h50m17.81574699s) [upload=5990]: Uploading segments... Aug 7 22:17:04.228721386 (5h50m17.815777607s) [upload=5990]: Got next segment. Inline: true Aug 7 22:17:04.228725082 (5h50m17.815781303s) [upload=5990]: Waiting for error group managing segments... Aug 7 22:17:04.228765939 (5h50m17.815822161s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:04.228772071 (5h50m17.815828292s) [upload=5990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:04.228775867 (5h50m17.815832089s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:04.215607888 (5h51m17.802664109s) [upload=5990]: Done uploading segments... Aug 7 22:18:04.215625421 (5h51m17.802681632s) [upload=5990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:04.292132983 (5h51m17.879189234s) [upload=6004]: Starting upload Aug 7 22:18:04.292228702 (5h51m17.879284943s) [upload=6004]: Done starting upload Aug 7 22:18:04.292318189 (5h51m17.879374439s) [upload=6004]: Uploading segments... Aug 7 22:18:04.292362020 (5h51m17.879418262s) [upload=6004]: Got next segment. Inline: true Aug 7 22:18:04.292367411 (5h51m17.879423662s) [upload=6004]: Waiting for error group managing segments... Aug 7 22:18:04.292410401 (5h51m17.879466651s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:04.292416432 (5h51m17.879472673s) [upload=6004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:04.292419948 (5h51m17.8794762s) [upload=6004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:04.275395564 (5h52m17.862451785s) [upload=6004]: Done uploading segments... Aug 7 22:19:04.275413478 (5h52m17.862469689s) [upload=6004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001054891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXX.X..................................XXXXXXXXXXXXXX] duration: 30.001097604s FAIL (2025-08-07 22:21:11 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001116549s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..X....................................................X....] duration: 112.876489ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:21:10 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 112.89848ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXX...X.....X..............................XXXXXXXXXXXX] duration: 30.001474385s FAIL (2025-08-07 22:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001531132s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)