Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.018621738s trace: https://jaeger.dev.storj.io/trace/3fba58ae710e8bb3 FAIL (2025-12-10 23:28:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Dec 10 23:27:19.681550102 (127h58m7.379118418s) [seg_pos={0 0}, upload=112343]: Begin upload segment... Dec 10 23:27:19.681581230 (127h58m7.379149546s) [seg_pos={0 0}, upload=112343]: Done begin upload segment. Dec 10 23:27:19.681619011 (127h58m7.379187318s) [noise=true , address=145.79.227.26:4682 , piece_id=RF3XDAZOFXKVSTF5HQJBAMPLQPI2LATYWWE6RBDDL3U3MKCHLIOQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.681627767 (127h58m7.379196084s) [noise=true , address=45.91.116.7:4025 , piece_id=EPINVLL3ITDXM4P25CBYVBPUCEHB7MGU272POY6VGYTS4Y3CKC5Q, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682040538 (127h58m7.379608845s) [noise=true , address=45.91.116.10:4107 , piece_id=CCJPHA7YKCYA6LGEO4MFJEM444HJQZYLONSSAL6VWBPRNS5F7J3Q, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682123012 (127h58m7.379691318s) [noise=true , address=77.90.31.19:4865 , piece_id=2AHBZRXC7TWWE54B7NMSCHQCWDOHQLOOBSPTJ5QKIHI7TUD6GL6A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682366848 (127h58m7.379935164s) [noise=true , address=64.49.13.169:4706 , piece_id=UPH2ZKJZQWM4WBMW4HV5ZEXKAM4SIVAHBD3X54IGQJV3KU5ZDAQQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682411221 (127h58m7.379979527s) [noise=true , address=45.91.116.11:4127 , piece_id=53RQOBTUYP2MA2MXSA6BKFN2E5ELO5NOLOBA5H5BRGQKPUSCLZLQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682706613 (127h58m7.380274929s) [noise=true , address=145.79.227.27:4710 , piece_id=7FBDRLN2SFMS4YYIZMKZFJZNCBL3NK5JP3OEZ7AM5YF2JEGK7QZA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682799836 (127h58m7.380368142s) [noise=true , address=77.90.31.17:4779 , piece_id=5X4LOQORHXU27QY2IHMUPKTRLDNQOMC2E3KTSAKVDUCZPKZ5ZK3A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682952782 (127h58m7.380521098s) [noise=true , address=77.90.31.12:4617 , piece_id=EHWCQDIJJ2TC6MUAUHVQLUQTANA5G322AVTSKWUET3AOQX6ERFMQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683155231 (127h58m7.380723537s) [noise=true , address=77.90.31.15:4726 , piece_id=6J2T4CQLERLF4UFH3R3O2OHZPAUHOI52WLOGC3G7LQPRV5OCAF6A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683197590 (127h58m7.380765895s) [noise=true , address=45.91.116.13:4183 , piece_id=YGPJWSOHPWYSOVKP6UMZS2Z26Y6TKINSE6JWHDIACF55DO4UMRZA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683431507 (127h58m7.380999813s) [noise=true , address=77.90.31.11:4585 , piece_id=VG4YW4PLQ6JN6WN3E63MGP56JQYCP5ODPCHFSFZLXGISQLPWRZYA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683539318 (127h58m7.381107624s) [noise=true , address=5.231.73.4:4003 , piece_id=63OP26YEX2NS2PXPHRZOSHCV75E4GTKIHHHOUA66FAFPHFGDFWOA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683675372 (127h58m7.381243679s) [noise=true , address=145.79.227.6:4269 , piece_id=PA4VKRYYLDN5IVZXPCK5YYXGJ4BUXL2UZXTINAKZHUS5764RAWBA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683886857 (127h58m7.381455163s) [noise=true , address=145.79.227.30:4771 , piece_id=EF5UEFUVPXMDRBRYP45V2BRVKOB2ADIYRDJ664ZHKIOBBHB3V7EA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683971396 (127h58m7.381539712s) [noise=true , address=77.90.31.16:4765 , piece_id=3OGZDEK2N4CNSQWBJVMCODS4QPENAY4ICDOXWKPOJOVGTDPFSNFA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.684226221 (127h58m7.381794527s) [noise=true , address=45.91.116.5:4323 , piece_id=EFCTT36ROZZ7IOLQOBC52ESK774YBSNQHQEA22JZIQZLKZ6TEQUA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.684487549 (127h58m7.382055855s) [noise=true , address=45.91.116.6:4996 , piece_id=5L2BAN466IC76CX44RFQCPTPFH4CETR4TTFHEU5JN3UIEKKCSSRQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.755764983 (127h58m7.453333288s) [upload=112343]: Next returned nil segment Dec 10 23:27:19.755773819 (127h58m7.453342125s) [upload=112343]: Waiting for error group managing segments... Dec 10 23:27:19.764164843 (127h58m7.461733159s) [noise=true , address=45.91.116.11:4127 , piece_id=53RQOBTUYP2MA2MXSA6BKFN2E5ELO5NOLOBA5H5BRGQKPUSCLZLQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.764652745 (127h58m7.462221051s) [noise=true , address=45.91.116.6:4996 , piece_id=5L2BAN466IC76CX44RFQCPTPFH4CETR4TTFHEU5JN3UIEKKCSSRQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.765027465 (127h58m7.462595771s) [noise=true , address=45.91.116.13:4183 , piece_id=YGPJWSOHPWYSOVKP6UMZS2Z26Y6TKINSE6JWHDIACF55DO4UMRZA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.769236768 (127h58m7.466805074s) [noise=true , address=145.79.227.27:4710 , piece_id=7FBDRLN2SFMS4YYIZMKZFJZNCBL3NK5JP3OEZ7AM5YF2JEGK7QZA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.771386334 (127h58m7.46895464s) [noise=true , address=45.91.116.5:4323 , piece_id=EFCTT36ROZZ7IOLQOBC52ESK774YBSNQHQEA22JZIQZLKZ6TEQUA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.777824189 (127h58m7.475392495s) [noise=true , address=45.91.116.10:4107 , piece_id=CCJPHA7YKCYA6LGEO4MFJEM444HJQZYLONSSAL6VWBPRNS5F7J3Q, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.782284811 (127h58m7.479853127s) [noise=true , address=145.79.227.26:4682 , piece_id=RF3XDAZOFXKVSTF5HQJBAMPLQPI2LATYWWE6RBDDL3U3MKCHLIOQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.783013341 (127h58m7.480581648s) [noise=true , address=45.91.116.7:4025 , piece_id=EPINVLL3ITDXM4P25CBYVBPUCEHB7MGU272POY6VGYTS4Y3CKC5Q, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.787877468 (127h58m7.485445784s) [noise=true , address=145.79.227.6:4269 , piece_id=PA4VKRYYLDN5IVZXPCK5YYXGJ4BUXL2UZXTINAKZHUS5764RAWBA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.863831357 (127h58m7.561399663s) [noise=true , address=5.231.73.4:4003 , piece_id=63OP26YEX2NS2PXPHRZOSHCV75E4GTKIHHHOUA66FAFPHFGDFWOA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:20.708076114 (127h58m8.40564445s) [noise=true , address=145.79.227.30:4771 , piece_id=EF5UEFUVPXMDRBRYP45V2BRVKOB2ADIYRDJ664ZHKIOBBHB3V7EA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:32.149149796 (127h58m19.846718112s) [noise=true , address=64.49.13.169:4706 , piece_id=UPH2ZKJZQWM4WBMW4HV5ZEXKAM4SIVAHBD3X54IGQJV3KU5ZDAQQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:28:19.644676307 (127h59m7.342244613s) [noise=true , address=77.90.31.16:4765 , piece_id=3OGZDEK2N4CNSQWBJVMCODS4QPENAY4ICDOXWKPOJOVGTDPFSNFA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: write tcp 5.161.89.252:8882->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644762338 (127h59m7.342330644s) [noise=true , address=77.90.31.12:4617 , piece_id=EHWCQDIJJ2TC6MUAUHVQLUQTANA5G322AVTSKWUET3AOQX6ERFMQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: write tcp 5.161.89.252:8210->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644829012 (127h59m7.342397328s) [noise=true , address=77.90.31.17:4779 , piece_id=5X4LOQORHXU27QY2IHMUPKTRLDNQOMC2E3KTSAKVDUCZPKZ5ZK3A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: rpc: dial tcp 77.90.31.17:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644888714 (127h59m7.34245702s) [noise=true , address=77.90.31.19:4865 , piece_id=2AHBZRXC7TWWE54B7NMSCHQCWDOHQLOOBSPTJ5QKIHI7TUD6GL6A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 5.161.89.252:11072->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.645022775 (127h59m7.342591091s) [noise=true , address=77.90.31.11:4585 , piece_id=VG4YW4PLQ6JN6WN3E63MGP56JQYCP5ODPCHFSFZLXGISQLPWRZYA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: write tcp 5.161.89.252:39502->77.90.31.11:4585: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.645194445 (127h59m7.342762761s) [noise=true , address=77.90.31.15:4726 , piece_id=6J2T4CQLERLF4UFH3R3O2OHZPAUHOI52WLOGC3G7LQPRV5OCAF6A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 5.161.89.252:27054->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:28:19.645233558 (127h59m7.342801864s) [seg_pos={0 0}, upload=112343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:28:19.661401509 (127h59m7.358969825s) [upload=112343]: Done uploading segments... Dec 10 23:28:19.661423810 (127h59m7.358992117s) [upload=112343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.018632889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 606.773803ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:27:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 606.789211ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................................XX.XX.....] duration: 479.202596ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:27:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.24685ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X..X.X.] duration: 378.797349ms trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:27:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.812979ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.023032945s trace: https://jaeger.dev.storj.io/trace/5c4d409e29f36c6c FAIL (2025-12-10 23:27:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== uestItem_ObjectBegin Dec 10 23:26:19.579841936 (127h57m7.277410273s) [upload=112324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:26:19.600543667 (127h57m7.298111972s) [seg_pos={0 0}, upload=112324]: Begin upload segment... Dec 10 23:26:19.600584852 (127h57m7.298153168s) [seg_pos={0 0}, upload=112324]: Done begin upload segment. Dec 10 23:26:19.600615400 (127h57m7.298183706s) [noise=true , address=77.90.31.17:4798 , piece_id=33MCTJ7D3RGLRFPIAE3335YJ5JBHQE4BV5OLT2KDXT3PUQKQO77Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.600654854 (127h57m7.29822319s) [noise=true , address=145.79.227.36:4533 , piece_id=ZHC42MNF33ATFJL76WZ333T5E2BWGJX4RMN4HATOJPXSY73NRRMQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601166870 (127h57m7.298735186s) [noise=true , address=77.90.31.16:4759 , piece_id=54IGT3HOBEGB3B4DEP43XQCZGJXS7NXRQWZIQ5Z4LLVWZV7EPPOQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601181628 (127h57m7.298749963s) [noise=true , address=5.231.73.2:4012 , piece_id=AJHGFXD66LC4JLSERUMDQITR6EVRENVJH6VFGLOQUMD7LHGDK5QA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601430613 (127h57m7.298998919s) [noise=true , address=145.79.227.28:4728 , piece_id=GZGVXWTHWEKWGTQAAIQZGRXTGWAL4RJZKXIF7VQCSY3BUA5KGMRA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601637520 (127h57m7.299205855s) [noise=true , address=45.91.116.13:4161 , piece_id=VQYSSMPZFZSBMZFMN65SQMSRQRHWES6HD3TDBRXCXY7LRSHZZ4LQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601692822 (127h57m7.299261138s) [noise=true , address=45.91.116.11:4136 , piece_id=ZWIB2CLSZH67GMO6J2CC72YTANT7CSWY4JBRA6OBTIFGEPQPSDQQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601929084 (127h57m7.2994974s) [noise=true , address=64.49.13.26:4953 , piece_id=H7N53TT5IN7UNYVRCDFPHR3KJUDVQKPZNFDL4ZGLNBCGK4MJT3QA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602040532 (127h57m7.299608878s) [noise=true , address=145.79.227.11:4752 , piece_id=BVXXW7W5P4IPAZDUV4YDCEKNAVCPBI7AB2DUHE3SFHY4JA4GDTFA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602177889 (127h57m7.299746205s) [noise=true , address=45.91.116.6:4983 , piece_id=NBDW5F7M2PPB2Z3PK5EVGWVVZIQNFH3G5BQOUJIZGCPUWQUMMVGA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602418538 (127h57m7.299986854s) [noise=true , address=145.79.227.21:4484 , piece_id=RMQ4P3IIOTS4TFU7RNKRDBAV4YLG53RY6WDH54TJL3A43IV6A7RA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602467991 (127h57m7.300036326s) [noise=true , address=64.49.13.154:4929 , piece_id=H5WIUBNV4YDNC7LWBGUZETUW746VWVVVD6R7HQLAMGSKUCNNT5HA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602682811 (127h57m7.300251117s) [noise=true , address=77.90.31.14:4695 , piece_id=KCUPUEFVMFMPCTMEQVCQB4R65CJYTPRFF2H4KYW43T7FIVZVZZZA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602870663 (127h57m7.300439009s) [noise=true , address=64.49.13.155:4913 , piece_id=SRJNPBJNTPWWSJSFVDJXIJRPOB4ZKMVJAWO5Q43P5IJULZGSB6CQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602921428 (127h57m7.300489744s) [noise=true , address=45.91.116.8:4062 , piece_id=UVAZCGUHD43CLY47C6SQBFQSGGRESKD7YLVBRMTCFEKTG46UDTYQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.603267955 (127h57m7.300836291s) [noise=true , address=77.90.31.11:4578 , piece_id=X4XP3W4N7JFGMNWJZTQDGJJALNY45BA4AH52OY2T6NVJGRFLVTIA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.603797844 (127h57m7.301366181s) [noise=true , address=77.90.31.19:4872 , piece_id=M5ZK7E6JF6QOAB5YIOUQTQFRKQNV52BPAS5CWIHY6IWH5ZHZNE2Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.604327254 (127h57m7.30189556s) [noise=true , address=77.90.31.21:4912 , piece_id=RKF4UR2UB2D7UDTF2LIQPPOOONKSQWDUTASMJQNBN4CEPQKNKD6Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.691888123 (127h57m7.38945643s) [upload=112324]: Next returned nil segment Dec 10 23:26:19.691899936 (127h57m7.389468242s) [upload=112324]: Waiting for error group managing segments... Dec 10 23:26:19.702414046 (127h57m7.399982382s) [noise=true , address=45.91.116.8:4062 , piece_id=UVAZCGUHD43CLY47C6SQBFQSGGRESKD7YLVBRMTCFEKTG46UDTYQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.703047660 (127h57m7.400615966s) [noise=true , address=45.91.116.11:4136 , piece_id=ZWIB2CLSZH67GMO6J2CC72YTANT7CSWY4JBRA6OBTIFGEPQPSDQQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.703598299 (127h57m7.401166605s) [noise=true , address=145.79.227.21:4484 , piece_id=RMQ4P3IIOTS4TFU7RNKRDBAV4YLG53RY6WDH54TJL3A43IV6A7RA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.704055682 (127h57m7.401623988s) [noise=true , address=145.79.227.36:4533 , piece_id=ZHC42MNF33ATFJL76WZ333T5E2BWGJX4RMN4HATOJPXSY73NRRMQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.704260034 (127h57m7.401828371s) [noise=true , address=45.91.116.6:4983 , piece_id=NBDW5F7M2PPB2Z3PK5EVGWVVZIQNFH3G5BQOUJIZGCPUWQUMMVGA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.706483678 (127h57m7.404051984s) [noise=true , address=145.79.227.11:4752 , piece_id=BVXXW7W5P4IPAZDUV4YDCEKNAVCPBI7AB2DUHE3SFHY4JA4GDTFA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.719169484 (127h57m7.41673783s) [noise=true , address=45.91.116.13:4161 , piece_id=VQYSSMPZFZSBMZFMN65SQMSRQRHWES6HD3TDBRXCXY7LRSHZZ4LQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.720469854 (127h57m7.41803816s) [noise=true , address=145.79.227.28:4728 , piece_id=GZGVXWTHWEKWGTQAAIQZGRXTGWAL4RJZKXIF7VQCSY3BUA5KGMRA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.837660411 (127h57m7.535228718s) [noise=true , address=5.231.73.2:4012 , piece_id=AJHGFXD66LC4JLSERUMDQITR6EVRENVJH6VFGLOQUMD7LHGDK5QA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.173210389 (127h57m7.870778705s) [noise=true , address=64.49.13.154:4929 , piece_id=H5WIUBNV4YDNC7LWBGUZETUW746VWVVVD6R7HQLAMGSKUCNNT5HA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.206097370 (127h57m7.903665686s) [noise=true , address=64.49.13.26:4953 , piece_id=H7N53TT5IN7UNYVRCDFPHR3KJUDVQKPZNFDL4ZGLNBCGK4MJT3QA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.209856212 (127h57m7.907424528s) [noise=true , address=64.49.13.155:4913 , piece_id=SRJNPBJNTPWWSJSFVDJXIJRPOB4ZKMVJAWO5Q43P5IJULZGSB6CQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:27:19.564917327 (127h58m7.262485632s) [noise=true , address=77.90.31.17:4798 , piece_id=33MCTJ7D3RGLRFPIAE3335YJ5JBHQE4BV5OLT2KDXT3PUQKQO77Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.564932515 (127h58m7.26250082s) [noise=true , address=77.90.31.21:4912 , piece_id=RKF4UR2UB2D7UDTF2LIQPPOOONKSQWDUTASMJQNBN4CEPQKNKD6Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: rpc: dial tcp 77.90.31.21:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565120857 (127h58m7.262689163s) [noise=true , address=77.90.31.16:4759 , piece_id=54IGT3HOBEGB3B4DEP43XQCZGJXS7NXRQWZIQ5Z4LLVWZV7EPPOQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565269625 (127h58m7.262837931s) [noise=true , address=77.90.31.14:4695 , piece_id=KCUPUEFVMFMPCTMEQVCQB4R65CJYTPRFF2H4KYW43T7FIVZVZZZA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:8202->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565496899 (127h58m7.263065205s) [noise=true , address=77.90.31.19:4872 , piece_id=M5ZK7E6JF6QOAB5YIOUQTQFRKQNV52BPAS5CWIHY6IWH5ZHZNE2Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 5.161.89.252:14264->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565567061 (127h58m7.263135366s) [noise=true , address=77.90.31.11:4578 , piece_id=X4XP3W4N7JFGMNWJZTQDGJJALNY45BA4AH52OY2T6NVJGRFLVTIA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: rpc: dial tcp 77.90.31.11:4578: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565602506 (127h58m7.263170822s) [seg_pos={0 0}, upload=112324]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:27:19.586667165 (127h58m7.284235471s) [upload=112324]: Done uploading segments... Dec 10 23:27:19.586686340 (127h58m7.284254647s) [upload=112324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.023043905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X..X.X] duration: 30.001346416s trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001362527s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................................XX.XX....] duration: 477.085862ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:27:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.115928ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 1.05512393s trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:27:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.055147875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................................XX.XX...] duration: 572.252953ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:26:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.288761ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 596.587076ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:26:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 596.60453ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.020056069s trace: https://jaeger.dev.storj.io/trace/32b660467c5855f7 FAIL (2025-12-10 23:26:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== segment... Dec 10 23:25:19.528626832 (127h56m7.226195148s) [seg_pos={0 0}, upload=112309]: Done begin upload segment. Dec 10 23:25:19.528647662 (127h56m7.226215978s) [noise=true , address=77.90.31.18:4813 , piece_id=3LBO4SOMOGIMWAQ7O3NPYAAKG6E5PMIUNJOJAS2CIGJXOAH322OA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.528657450 (127h56m7.226225756s) [noise=true , address=5.231.73.4:4008 , piece_id=MN4ITDTRGKI3H5E4XGY6OYKHADF5VC4HOWQLE7Y6IBT7STEIQVHQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529001602 (127h56m7.226569909s) [noise=true , address=45.91.116.9:4294 , piece_id=HCJU5F4V2YEYC5KUTCLENM2LQ37SIPEQORYYI7D7PNYZL4MM3MHQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529137185 (127h56m7.226705501s) [noise=true , address=145.79.227.29:4962 , piece_id=GKFHWDWZV3PR4BXJW56W4FPVBTZK3SPGXFZBCIB6DEWNKL7LVOXA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529315770 (127h56m7.226884075s) [noise=true , address=45.91.116.8:4060 , piece_id=COTKSF6CA4XMJBTN3NWEXKDDDJSRBUE5EUE5EKUOM4ZBBPKMWT6Q, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529395239 (127h56m7.226963544s) [noise=true , address=45.91.116.11:4144 , piece_id=SI4GNJDSHZQP5MUZK2QHQOOH64TASBW3LKYZWCGDG4WGA6CTJPAQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529656767 (127h56m7.227225083s) [noise=true , address=77.90.31.19:4860 , piece_id=HBYLLEDWXPB4CCLVSVG37BX2NPDBMKF7MERGURRVNNN76HGOO2LA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529734341 (127h56m7.227302647s) [noise=true , address=77.90.31.21:4898 , piece_id=LXPOHPF2CKJBKBUCX7NDOOFY362KH7ATPHTD2IX45ZBLV4MJ3XOQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.529898098 (127h56m7.227466403s) [noise=true , address=77.90.31.11:4588 , piece_id=HETULWG4XX7GU2MNFJ6E6RFSZWYF74NB2PSDAMQEZL6UEUFJDUNQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.530057856 (127h56m7.227626172s) [noise=true , address=77.90.31.15:4719 , piece_id=USXNEBVGIHN2ON434KQHTIDQ57WU7RIPJ5FRN2O6IKFG5PG3TVEQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.530127776 (127h56m7.227696092s) [noise=true , address=145.79.227.48:4312 , piece_id=WLTY54QEKCI2QEALRN5EAUDMCFWXKDPTKOCXXYJ2XKGMHWK57MUQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.530368927 (127h56m7.227937233s) [noise=true , address=45.91.116.5:4305 , piece_id=W7HXJWGUMXDWZLALLMF34T6YKDS5PHYJY4OEW6L4GZ7ZLBAJ7S4Q, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.530727907 (127h56m7.228296223s) [noise=true , address=77.90.31.13:4639 , piece_id=BN3Q2LWXWPVCTDABOWRCQ6MZLVZXR26HFOVAL6OHBGCJZM36ORFQ, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.531080045 (127h56m7.228648361s) [noise=true , address=64.49.13.169:4707 , piece_id=6Z7SYIGUASLH27PXUPNS3SMOEFSEWDCFLEZTY7N27T3MCWT5YPGA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.531197906 (127h56m7.228766222s) [noise=true , address=45.91.116.10:4089 , piece_id=IEYIMMKWGNRESKWMMO4TEM6C7ISTRK5E5Y6BPLQUT5E2JKCJXGOA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.531469453 (127h56m7.229037769s) [noise=true , address=77.90.31.10:4545 , piece_id=WI7I57RQR5TRTHRSXHS24DQP37PHXNSCF25AIVILN52EBTA7TN4A, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.531547819 (127h56m7.229116125s) [noise=true , address=145.79.227.15:4897 , piece_id=N5KSQ4D5LI2W6KRS4AHJSKNYOUPMEJJT663PRLFJLS3VKNEB2HFA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.532385824 (127h56m7.229954141s) [noise=true , address=145.79.227.33:4903 , piece_id=KMSXROPIWG6VM2G7RLNTQJVCSPNNAVSCR3TC2M3HETJQJ5H2UMPA, seg_pos={0 0}, upload=112309]: Uploading piece... Dec 10 23:25:19.602124054 (127h56m7.29969237s) [upload=112309]: Next returned nil segment Dec 10 23:25:19.602132689 (127h56m7.299700996s) [upload=112309]: Waiting for error group managing segments... Dec 10 23:25:19.610825778 (127h56m7.308394084s) [noise=true , address=45.91.116.8:4060 , piece_id=COTKSF6CA4XMJBTN3NWEXKDDDJSRBUE5EUE5EKUOM4ZBBPKMWT6Q, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.612018336 (127h56m7.309586641s) [noise=true , address=45.91.116.11:4144 , piece_id=SI4GNJDSHZQP5MUZK2QHQOOH64TASBW3LKYZWCGDG4WGA6CTJPAQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.612847875 (127h56m7.310416192s) [noise=true , address=145.79.227.48:4312 , piece_id=WLTY54QEKCI2QEALRN5EAUDMCFWXKDPTKOCXXYJ2XKGMHWK57MUQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.614438336 (127h56m7.312006652s) [noise=true , address=145.79.227.33:4903 , piece_id=KMSXROPIWG6VM2G7RLNTQJVCSPNNAVSCR3TC2M3HETJQJ5H2UMPA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.631799266 (127h56m7.329367602s) [noise=true , address=45.91.116.9:4294 , piece_id=HCJU5F4V2YEYC5KUTCLENM2LQ37SIPEQORYYI7D7PNYZL4MM3MHQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.634374647 (127h56m7.331942973s) [noise=true , address=45.91.116.5:4305 , piece_id=W7HXJWGUMXDWZLALLMF34T6YKDS5PHYJY4OEW6L4GZ7ZLBAJ7S4Q, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.636377367 (127h56m7.333945703s) [noise=true , address=45.91.116.10:4089 , piece_id=IEYIMMKWGNRESKWMMO4TEM6C7ISTRK5E5Y6BPLQUT5E2JKCJXGOA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.657547312 (127h56m7.355115619s) [noise=true , address=145.79.227.15:4897 , piece_id=N5KSQ4D5LI2W6KRS4AHJSKNYOUPMEJJT663PRLFJLS3VKNEB2HFA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.753975617 (127h56m7.451543954s) [noise=true , address=5.231.73.4:4008 , piece_id=MN4ITDTRGKI3H5E4XGY6OYKHADF5VC4HOWQLE7Y6IBT7STEIQVHQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:19.754157918 (127h56m7.451726224s) [noise=true , address=145.79.227.29:4962 , piece_id=GKFHWDWZV3PR4BXJW56W4FPVBTZK3SPGXFZBCIB6DEWNKL7LVOXA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:25:25.289158006 (127h56m12.986726322s) [noise=true , address=64.49.13.169:4707 , piece_id=6Z7SYIGUASLH27PXUPNS3SMOEFSEWDCFLEZTY7N27T3MCWT5YPGA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: <nil> Dec 10 23:26:19.495373402 (127h57m7.192941708s) [noise=true , address=77.90.31.19:4860 , piece_id=HBYLLEDWXPB4CCLVSVG37BX2NPDBMKF7MERGURRVNNN76HGOO2LA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, address:77.90.31.19:4860): protocol: rpc: dial tcp 77.90.31.19:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.495549912 (127h57m7.193118228s) [noise=true , address=77.90.31.15:4719 , piece_id=USXNEBVGIHN2ON434KQHTIDQ57WU7RIPJ5FRN2O6IKFG5PG3TVEQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk, address:77.90.31.15:4719): protocol: rpc: dial tcp 77.90.31.15:4719: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.495690485 (127h57m7.193258801s) [noise=true , address=77.90.31.18:4813 , piece_id=3LBO4SOMOGIMWAQ7O3NPYAAKG6E5PMIUNJOJAS2CIGJXOAH322OA, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: rpc: dial tcp 77.90.31.18:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.495798496 (127h57m7.193366832s) [noise=true , address=77.90.31.10:4545 , piece_id=WI7I57RQR5TRTHRSXHS24DQP37PHXNSCF25AIVILN52EBTA7TN4A, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.495857566 (127h57m7.193425872s) [noise=true , address=77.90.31.11:4588 , piece_id=HETULWG4XX7GU2MNFJ6E6RFSZWYF74NB2PSDAMQEZL6UEUFJDUNQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: write tcp 5.161.89.252:26686->77.90.31.11:4588: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.496076486 (127h57m7.193644831s) [noise=true , address=77.90.31.13:4639 , piece_id=BN3Q2LWXWPVCTDABOWRCQ6MZLVZXR26HFOVAL6OHBGCJZM36ORFQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: write tcp 5.161.89.252:11128->77.90.31.13:4639: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.496176091 (127h57m7.193744437s) [noise=true , address=77.90.31.21:4898 , piece_id=LXPOHPF2CKJBKBUCX7NDOOFY362KH7ATPHTD2IX45ZBLV4MJ3XOQ, seg_pos={0 0}, upload=112309]: Done uploading piece. err: ecclient: upload failed (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx, address:77.90.31.21:4898): protocol: rpc: dial tcp 77.90.31.21:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:19.496203683 (127h57m7.193772029s) [seg_pos={0 0}, upload=112309]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:26:19.513794091 (127h57m7.211362407s) [upload=112309]: Done uploading segments... Dec 10 23:26:19.513812315 (127h57m7.211380621s) [upload=112309]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.020065406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................................................XX.XX..] duration: 1.323093331s trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:26:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.323117335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 402.640501ms trace: https://jaeger.dev.storj.io/trace/387fbdb43f812d66 FAIL (2025-12-10 23:25:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 402.655069ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................................................XX.XX.] duration: 470.459954ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:25:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.477968ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 576.861539ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:25:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.876497ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.000542293s trace: https://jaeger.dev.storj.io/trace/387fbdb43f812d66 FAIL (2025-12-10 23:25:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000556761s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001218579s trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:25:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001226153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.016115299s trace: https://jaeger.dev.storj.io/trace/37f7ddfef60bb8c2 FAIL (2025-12-10 23:25:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:24:19.439050671 (127h55m7.136618977s) [upload=112295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:24:19.458786728 (127h55m7.156355023s) [seg_pos={0 0}, upload=112295]: Begin upload segment... Dec 10 23:24:19.458830819 (127h55m7.156399125s) [seg_pos={0 0}, upload=112295]: Done begin upload segment. Dec 10 23:24:19.458875764 (127h55m7.156444059s) [noise=true , address=145.79.227.30:4769 , piece_id=HXEBHKH5G75XR7I7ZZSLU2UURVV5SGG6ZFHO6BXHXTWBRIMNBNYA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.458900600 (127h55m7.156468906s) [noise=true , address=77.90.31.11:4576 , piece_id=5BKEPQZRWRTCT7CLDMXDU2C5UGNVA7URXS3PSDANNAM75QVKFGKQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.459218423 (127h55m7.156786729s) [noise=true , address=45.91.116.6:4996 , piece_id=OCOUVO63FVYZMKIAO7Q27P7PR2AH5EERCNFTTKQJULVJRHCQLPEA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.459547869 (127h55m7.157116175s) [noise=true , address=45.91.116.13:4165 , piece_id=C5SM2IGNO4NKKRJIVZD7YPDABQBCNVFYAKU5EK3VLVINQWWX445Q, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.459942816 (127h55m7.157511123s) [noise=true , address=145.79.227.29:4951 , piece_id=U2FFTCIVL54MABK6CIAT5XXC2LFSU5VGXXVFV2OXPFTF7GF3BPDA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.460285507 (127h55m7.157853803s) [noise=true , address=145.79.227.10:4421 , piece_id=A3HUAWRUTBLR6XMRWOFKFLPKWRKYNRPMPOYIS7RHNVHY3P6SXKJA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.460668081 (127h55m7.158236388s) [noise=true , address=45.91.116.12:4238 , piece_id=3OHKVI7DS47MR3PXMJBYOYLM2VBZNX5EDL4XSBYF6EHD6LL3QQTA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.461024508 (127h55m7.158592814s) [noise=true , address=77.90.31.11:4557 , piece_id=NWQY554XJUPJXGNXEQX44XFJMPKX3DF5IIOYRO3WPKA5VY3TCD6A, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.461402484 (127h55m7.158970779s) [noise=true , address=77.90.31.17:4774 , piece_id=WRV7TKSVL44GPCFZRXI2NRKFSRBP73PVN7KEFRU35JOLNUPVUYMQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.461778386 (127h55m7.159346691s) [noise=true , address=45.91.116.5:4309 , piece_id=6SHWBY7VCBJM72FZ2PV5WA3UBLILGNUSK766SFD7SK2KSXDHF7FQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.462100577 (127h55m7.159668883s) [noise=true , address=77.90.31.13:4658 , piece_id=JG3HD4FZ7WYM2LLJIDSVP4RXMBEGO27XV6XS4YNGZXHB7SRKOGYQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.462424182 (127h55m7.159992488s) [noise=true , address=45.91.116.14:4212 , piece_id=BPPKCY4KV6GC4QFKGW3AW7XZGJS736JCESLWQBG4H4ZTGYFAPC7A, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.462720845 (127h55m7.160289152s) [noise=true , address=77.90.31.12:4604 , piece_id=TNMES6NV7XECSHFHXN7CGXNSUNST6HL5R72P5AK6YDFTWTA4K4WQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.462976854 (127h55m7.160545161s) [noise=true , address=145.79.227.32:4860 , piece_id=4BDCGGLDLIKRWHAVACNLH4FUQT6GMZLXBWBO54A7U3IWE4HMCDGA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.463229827 (127h55m7.160798133s) [noise=true , address=77.90.31.16:4749 , piece_id=JX6HMJEKMLK4CXQX7VGSL3M5T2Q6KL62KLVS6LPXUU4WRQZ77EFQ, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.463498909 (127h55m7.161067215s) [noise=true , address=45.91.116.7:4027 , piece_id=6EIMAW3BMI6M4Y752EQ7C45RUEHA3QNXZWWNGD437EQWJEIZC5ZA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.463772771 (127h55m7.161341077s) [noise=true , address=193.149.164.143:4695 , piece_id=SGYMFXHTCHDCCHZL2KH3O5K4VXVL54ZDEVD2KKDOS3EQ6RED5OFA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.464886291 (127h55m7.162454597s) [noise=true , address=5.231.73.5:4002 , piece_id=QDVR7BNPDSMVHGSIKFVVPOVHGQXRHZNBPX4T7IWELWRZO33A6KLA, seg_pos={0 0}, upload=112295]: Uploading piece... Dec 10 23:24:19.522292092 (127h55m7.219860398s) [upload=112295]: Next returned nil segment Dec 10 23:24:19.522299576 (127h55m7.219867883s) [upload=112295]: Waiting for error group managing segments... Dec 10 23:24:19.531416267 (127h55m7.228984572s) [noise=true , address=45.91.116.13:4165 , piece_id=C5SM2IGNO4NKKRJIVZD7YPDABQBCNVFYAKU5EK3VLVINQWWX445Q, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.532762250 (127h55m7.230330546s) [noise=true , address=45.91.116.12:4238 , piece_id=3OHKVI7DS47MR3PXMJBYOYLM2VBZNX5EDL4XSBYF6EHD6LL3QQTA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.533379193 (127h55m7.230947499s) [noise=true , address=45.91.116.6:4996 , piece_id=OCOUVO63FVYZMKIAO7Q27P7PR2AH5EERCNFTTKQJULVJRHCQLPEA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.534814745 (127h55m7.232383051s) [noise=true , address=145.79.227.10:4421 , piece_id=A3HUAWRUTBLR6XMRWOFKFLPKWRKYNRPMPOYIS7RHNVHY3P6SXKJA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.539365054 (127h55m7.23693336s) [noise=true , address=45.91.116.5:4309 , piece_id=6SHWBY7VCBJM72FZ2PV5WA3UBLILGNUSK766SFD7SK2KSXDHF7FQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.539585457 (127h55m7.237153773s) [noise=true , address=145.79.227.32:4860 , piece_id=4BDCGGLDLIKRWHAVACNLH4FUQT6GMZLXBWBO54A7U3IWE4HMCDGA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.543808955 (127h55m7.241377261s) [noise=true , address=45.91.116.14:4212 , piece_id=BPPKCY4KV6GC4QFKGW3AW7XZGJS736JCESLWQBG4H4ZTGYFAPC7A, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.544839461 (127h55m7.242407777s) [noise=true , address=45.91.116.7:4027 , piece_id=6EIMAW3BMI6M4Y752EQ7C45RUEHA3QNXZWWNGD437EQWJEIZC5ZA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.562254602 (127h55m7.259822908s) [noise=true , address=145.79.227.29:4951 , piece_id=U2FFTCIVL54MABK6CIAT5XXC2LFSU5VGXXVFV2OXPFTF7GF3BPDA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:19.650764304 (127h55m7.348332629s) [noise=true , address=5.231.73.5:4002 , piece_id=QDVR7BNPDSMVHGSIKFVVPOVHGQXRHZNBPX4T7IWELWRZO33A6KLA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:20.026562279 (127h55m7.724130595s) [noise=true , address=193.149.164.143:4695 , piece_id=SGYMFXHTCHDCCHZL2KH3O5K4VXVL54ZDEVD2KKDOS3EQ6RED5OFA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:24:20.527366676 (127h55m8.224934992s) [noise=true , address=145.79.227.30:4769 , piece_id=HXEBHKH5G75XR7I7ZZSLU2UURVV5SGG6ZFHO6BXHXTWBRIMNBNYA, seg_pos={0 0}, upload=112295]: Done uploading piece. err: <nil> Dec 10 23:25:19.425260077 (127h56m7.122828413s) [noise=true , address=77.90.31.11:4557 , piece_id=NWQY554XJUPJXGNXEQX44XFJMPKX3DF5IIOYRO3WPKA5VY3TCD6A, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: rpc: dial tcp 77.90.31.11:4557: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425295213 (127h56m7.122863549s) [noise=true , address=77.90.31.16:4749 , piece_id=JX6HMJEKMLK4CXQX7VGSL3M5T2Q6KL62KLVS6LPXUU4WRQZ77EFQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425536393 (127h56m7.123104729s) [noise=true , address=77.90.31.17:4774 , piece_id=WRV7TKSVL44GPCFZRXI2NRKFSRBP73PVN7KEFRU35JOLNUPVUYMQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: rpc: dial tcp 77.90.31.17:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425603970 (127h56m7.123172306s) [noise=true , address=77.90.31.11:4576 , piece_id=5BKEPQZRWRTCT7CLDMXDU2C5UGNVA7URXS3PSDANNAM75QVKFGKQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: rpc: dial tcp 77.90.31.11:4576: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425739162 (127h56m7.123307499s) [noise=true , address=77.90.31.13:4658 , piece_id=JG3HD4FZ7WYM2LLJIDSVP4RXMBEGO27XV6XS4YNGZXHB7SRKOGYQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425789406 (127h56m7.123357743s) [noise=true , address=77.90.31.12:4604 , piece_id=TNMES6NV7XECSHFHXN7CGXNSUNST6HL5R72P5AK6YDFTWTA4K4WQ, seg_pos={0 0}, upload=112295]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 5.161.89.252:55106->77.90.31.12:4604: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:19.425816105 (127h56m7.123384442s) [seg_pos={0 0}, upload=112295]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:25:19.440139331 (127h56m7.137707638s) [upload=112295]: Done uploading segments... Dec 10 23:25:19.440154911 (127h56m7.137723217s) [upload=112295]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.016124386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 335.423324ms trace: https://jaeger.dev.storj.io/trace/1f78af2cdccee82f FAIL (2025-12-10 23:24:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 335.438141ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................................................XX.XX] duration: 5.000479749s trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:24:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000495438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000713078s trace: https://jaeger.dev.storj.io/trace/152916cd220ea451 FAIL (2025-12-10 23:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000722525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.018215448s trace: https://jaeger.dev.storj.io/trace/6db07c8c4ac770f2 FAIL (2025-12-10 23:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== upload segment... Dec 10 23:23:19.390828382 (127h54m7.088396688s) [seg_pos={0 0}, upload=112282]: Done begin upload segment. Dec 10 23:23:19.390857396 (127h54m7.088425703s) [noise=true , address=5.231.73.4:4025 , piece_id=OFT2WPNFLIAV22EGO5NDVBEGMCJQO4GLJUIF24T6ULAN3OYMWMVQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.390867035 (127h54m7.088435351s) [noise=true , address=77.90.31.16:4746 , piece_id=ZRODXEQ6RHTGRG2KVT4TEBZ5ABORQRHSDHXCM5RTAS5JECU7OUHA, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.391196730 (127h54m7.088765046s) [noise=true , address=193.149.164.143:4698 , piece_id=VJQGP6DIAHMHF5BWCLQAMAKJG66VLMUJET7U6EBZY2345XEM2UNQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.391592689 (127h54m7.089161005s) [noise=true , address=145.79.227.22:4527 , piece_id=JUGZB2W2MIU4SESIOBAKG5XRCE2QODWGXJZ27K4VW3RDJCT7RK2A, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.391852625 (127h54m7.089420941s) [noise=true , address=45.91.116.9:4293 , piece_id=SINTWUBK5YZSBQV5CC2AVDDVDO3Z4XHN6IRYY6DQABEEINGGKXDQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.392261870 (127h54m7.089830186s) [noise=true , address=77.90.31.21:4880 , piece_id=F3JST7VURL32NPNZE4WUOBJCJLPP445CPPFCKFP2GQOKX2HCG5BQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.392681354 (127h54m7.09024967s) [noise=true , address=145.79.227.30:4788 , piece_id=M54WTNUBW6M35GYXZ4BRIEF4NDQJN4QDTCIHBW7Y5ZJG3UZUEIDQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.393102540 (127h54m7.090670847s) [noise=true , address=77.90.31.17:4784 , piece_id=XD3TOJVOAT56PVJ5XDUAZPBXB7ABIA42BTSHBMWV3FYV76LRHSYA, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.393354020 (127h54m7.090922336s) [noise=true , address=77.90.31.13:4640 , piece_id=2PTLL7IIEUJMBOKAFDLOUHU4DLWQEEOFVQULIY3S5YOZLREVH36A, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.393757313 (127h54m7.091325629s) [noise=true , address=145.79.227.44:4222 , piece_id=EGC3TQPFE4WB4HUFGS4BBZNKQBLIJWSQZ26UFECYWRJRNNT3DAGA, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.395404459 (127h54m7.092972766s) [noise=true , address=45.91.116.8:4064 , piece_id=FJHR6LWPB2AZ2FOVYYCISQTYH3GTMJCNVCMWOTLH4N63SH7POT2Q, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.396568084 (127h54m7.094136389s) [noise=true , address=45.91.116.14:4209 , piece_id=YYCM4PEXGFS5ZGQTJV6ECPJZLOJICBMJSHII4KDV55XB4XTDRZ2Q, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.397699317 (127h54m7.095267623s) [noise=true , address=77.90.31.19:4867 , piece_id=QD4CGAZ4VKBZPI5BDKIVWKDKZH2WRWWSUAIZZXY3W6RGNRWHNFOQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.398746473 (127h54m7.096314778s) [noise=true , address=77.90.31.10:4551 , piece_id=QX6XQNMU7YYQG2II4L6E5AUCUCTBH3Z22DA5JQDAYYQ2NRUZGCCQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.399778801 (127h54m7.097347117s) [noise=true , address=77.90.31.21:4886 , piece_id=6M626JT2A3MQZULCV3NMPHDTJIGQPDJEEJDB2WNQPYVXVRF4JPRQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.400869699 (127h54m7.098438015s) [noise=true , address=45.91.116.5:4307 , piece_id=RU32N7AVEKPOBSCF5AFRRQIGUFTYVZZ33SMQ3KGYHGJ3465IK6YQ, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.401965715 (127h54m7.099534022s) [noise=true , address=145.79.227.28:4747 , piece_id=6XDK5AP5CHGR3ZFDA5UBRG2QLXA56POSCJ4CR4A3IFIFCLHUIY6A, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.403096177 (127h54m7.100664484s) [noise=true , address=64.49.13.137:4504 , piece_id=CEWUMVNKMZIH4EZY4FVAHXM56KMSOG7JI3ECF5I232UDTXT34RZA, seg_pos={0 0}, upload=112282]: Uploading piece... Dec 10 23:23:19.466738568 (127h54m7.164306875s) [upload=112282]: Next returned nil segment Dec 10 23:23:19.466748998 (127h54m7.164317314s) [upload=112282]: Waiting for error group managing segments... Dec 10 23:23:19.475830662 (127h54m7.173398978s) [noise=true , address=45.91.116.8:4064 , piece_id=FJHR6LWPB2AZ2FOVYYCISQTYH3GTMJCNVCMWOTLH4N63SH7POT2Q, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.476687002 (127h54m7.174255318s) [noise=true , address=145.79.227.22:4527 , piece_id=JUGZB2W2MIU4SESIOBAKG5XRCE2QODWGXJZ27K4VW3RDJCT7RK2A, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.478958093 (127h54m7.176526399s) [noise=true , address=145.79.227.28:4747 , piece_id=6XDK5AP5CHGR3ZFDA5UBRG2QLXA56POSCJ4CR4A3IFIFCLHUIY6A, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.489215604 (127h54m7.18678392s) [noise=true , address=45.91.116.9:4293 , piece_id=SINTWUBK5YZSBQV5CC2AVDDVDO3Z4XHN6IRYY6DQABEEINGGKXDQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.489521406 (127h54m7.187089712s) [noise=true , address=45.91.116.14:4209 , piece_id=YYCM4PEXGFS5ZGQTJV6ECPJZLOJICBMJSHII4KDV55XB4XTDRZ2Q, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.496386259 (127h54m7.193954564s) [noise=true , address=45.91.116.5:4307 , piece_id=RU32N7AVEKPOBSCF5AFRRQIGUFTYVZZ33SMQ3KGYHGJ3465IK6YQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.584574019 (127h54m7.282142325s) [noise=true , address=5.231.73.4:4025 , piece_id=OFT2WPNFLIAV22EGO5NDVBEGMCJQO4GLJUIF24T6ULAN3OYMWMVQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.961401223 (127h54m7.658969529s) [noise=true , address=64.49.13.137:4504 , piece_id=CEWUMVNKMZIH4EZY4FVAHXM56KMSOG7JI3ECF5I232UDTXT34RZA, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:19.999808415 (127h54m7.697376732s) [noise=true , address=193.149.164.143:4698 , piece_id=VJQGP6DIAHMHF5BWCLQAMAKJG66VLMUJET7U6EBZY2345XEM2UNQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:20.399605507 (127h54m8.097173833s) [noise=true , address=145.79.227.30:4788 , piece_id=M54WTNUBW6M35GYXZ4BRIEF4NDQJN4QDTCIHBW7Y5ZJG3UZUEIDQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:23:20.438793147 (127h54m8.136361473s) [noise=true , address=145.79.227.44:4222 , piece_id=EGC3TQPFE4WB4HUFGS4BBZNKQBLIJWSQZ26UFECYWRJRNNT3DAGA, seg_pos={0 0}, upload=112282]: Done uploading piece. err: <nil> Dec 10 23:24:19.357463509 (127h55m7.055031855s) [noise=true , address=77.90.31.10:4551 , piece_id=QX6XQNMU7YYQG2II4L6E5AUCUCTBH3Z22DA5JQDAYYQ2NRUZGCCQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr, address:77.90.31.10:4551): protocol: rpc: dial tcp 77.90.31.10:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.357647473 (127h55m7.055215779s) [noise=true , address=77.90.31.16:4746 , piece_id=ZRODXEQ6RHTGRG2KVT4TEBZ5ABORQRHSDHXCM5RTAS5JECU7OUHA, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.357722362 (127h55m7.055290668s) [noise=true , address=77.90.31.13:4640 , piece_id=2PTLL7IIEUJMBOKAFDLOUHU4DLWQEEOFVQULIY3S5YOZLREVH36A, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: write tcp 5.161.89.252:22490->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.357804146 (127h55m7.055372452s) [noise=true , address=77.90.31.17:4784 , piece_id=XD3TOJVOAT56PVJ5XDUAZPBXB7ABIA42BTSHBMWV3FYV76LRHSYA, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: rpc: dial tcp 77.90.31.17:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.357951140 (127h55m7.055519446s) [noise=true , address=77.90.31.21:4886 , piece_id=6M626JT2A3MQZULCV3NMPHDTJIGQPDJEEJDB2WNQPYVXVRF4JPRQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: rpc: dial tcp 77.90.31.21:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.358083267 (127h55m7.055651583s) [noise=true , address=77.90.31.21:4880 , piece_id=F3JST7VURL32NPNZE4WUOBJCJLPP445CPPFCKFP2GQOKX2HCG5BQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, address:77.90.31.21:4880): protocol: write tcp 5.161.89.252:27748->77.90.31.21:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.358179126 (127h55m7.055747432s) [noise=true , address=77.90.31.19:4867 , piece_id=QD4CGAZ4VKBZPI5BDKIVWKDKZH2WRWWSUAIZZXY3W6RGNRWHNFOQ, seg_pos={0 0}, upload=112282]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: rpc: dial tcp 77.90.31.19:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:19.358205846 (127h55m7.055774162s) [seg_pos={0 0}, upload=112282]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:24:19.372926694 (127h55m7.070495011s) [upload=112282]: Done uploading segments... Dec 10 23:24:19.372947303 (127h55m7.070515619s) [upload=112282]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.018225246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................................XX.X] duration: 5.002781172s trace: https://jaeger.dev.storj.io/trace/73df0f646c7616b8 FAIL (2025-12-10 23:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00279576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 414.188601ms trace: https://jaeger.dev.storj.io/trace/1f78af2cdccee82f FAIL (2025-12-10 23:23:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 414.205753ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001348432s trace: https://jaeger.dev.storj.io/trace/758bc1790b9da9b6 FAIL (2025-12-10 23:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001369171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001153473s trace: https://jaeger.dev.storj.io/trace/1f78af2cdccee82f FAIL (2025-12-10 23:23:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001165716s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 598.638476ms trace: https://jaeger.dev.storj.io/trace/2637a749f06dbf5 FAIL (2025-12-10 23:23:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 598.651178ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.016659514s trace: https://jaeger.dev.storj.io/trace/2e426ed2e53efa07 FAIL (2025-12-10 23:23:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload segment... Dec 10 23:22:19.308384425 (127h53m7.005952731s) [seg_pos={0 0}, upload=112269]: Done begin upload segment. Dec 10 23:22:19.308404433 (127h53m7.005972729s) [noise=true , address=64.49.13.139:4473 , piece_id=OAMM5RBJAW4ICUCPG4YZJZSXGVJACZE6OKQ4D65PHVVNIFWWJYTQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.308510431 (127h53m7.006078767s) [noise=true , address=77.90.31.11:4576 , piece_id=Y6PZNRDF2XLI3YHXR5D6TYD2TTR665L3F4KIHRLSI5YIK4D4HTMA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.309032797 (127h53m7.006601133s) [noise=true , address=77.90.31.10:4536 , piece_id=ZTIQDNF7POKXV7SPJJEY4KYMPBIYPKPCFEJH6ODFL2L5EXZ6DR3Q, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.309557427 (127h53m7.007125763s) [noise=true , address=77.90.31.12:4601 , piece_id=QR3XV27RH4K5DAF434KZTRRTTSVRAX6POF52IINRWLVB7WKAVXKA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.310097024 (127h53m7.007665361s) [noise=true , address=45.91.116.12:4259 , piece_id=HCJAMWQHPOPL7KTV25OGFY4WWQ6TEZ4ZPNNC5V7OSBRS25OCKD2Q, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.310560329 (127h53m7.008128666s) [noise=true , address=145.79.227.42:4152 , piece_id=3ZTLZDEW3OHVL2CO4FJFLLQ4XEOGJLD76TBLYTAH2PWHO4DNUNYA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.311142447 (127h53m7.008710783s) [noise=true , address=77.90.31.17:4790 , piece_id=4ZH7QCL6INDT7OF2XOQVYSFZ3IEC5OOCBESWJ5P3AVXQF3JUFQBA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.311601474 (127h53m7.009169811s) [noise=true , address=77.90.31.13:4652 , piece_id=O2PYHR2UO6ZJZ7PZ7EMUEZHGC35QDWHSYMKGSQQB3R77QUZQNMYQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.312278670 (127h53m7.009847005s) [noise=true , address=45.91.116.13:4158 , piece_id=UHS7QZKTMBNF6SLE2EGSMZRI35CRG6H6I6SA2M5FKUFC3JL3YX2Q, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.313008733 (127h53m7.01057707s) [noise=true , address=145.79.227.17:4986 , piece_id=PD7TVILVPBO62NG4PZZRWV25WRDZAAT2W6CLZTLKDX4XSUZE4WVQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.313830107 (127h53m7.011398414s) [noise=true , address=145.79.227.24:4594 , piece_id=YAZHOUIDI4R5OOKOWVHWF6CXMRJTQTZGNAG7RKBXHPVZ3F7M3RLA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.315114417 (127h53m7.012682743s) [noise=true , address=193.149.164.143:4696 , piece_id=NLC3MRZEAXPWIS7VS3ARQPJENDRYIJ5LQAACRNFQEOXI3FU5PSNA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.316408924 (127h53m7.01397723s) [noise=true , address=145.79.227.33:4890 , piece_id=P3PXFVQ6HGZZLQ6PO4LNYT7YYBVB3QMRGTVMR24PYNDKCSPDXWLQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.318692691 (127h53m7.016261026s) [noise=true , address=77.90.31.14:4696 , piece_id=SOVODD7CSFQ3BS5ACB6KY7V6M7KWSFMYBUE4NSAMX6UUTWK36MNQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.319785271 (127h53m7.017353608s) [noise=true , address=77.90.31.21:4887 , piece_id=7A3D2GSN4MFQZJ5ID4S7PUHXEVGEO3NUH7A7WOQHP3RWW6TBU4IA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.326909228 (127h53m7.024477534s) [noise=true , address=145.79.227.12:4772 , piece_id=7HZPDMV4TPK3DSN42RIGQU7JHIHI43WHODVZAMKQ4AEEJ6QSNB6Q, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.327298035 (127h53m7.024866372s) [noise=true , address=45.91.116.11:4120 , piece_id=VGO5RNFPIQMUQCOADYBVY4BDHAU7W7DG7CYZPYIFVXWX4RIOJWIQ, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.328214006 (127h53m7.025782352s) [noise=true , address=45.91.116.5:4325 , piece_id=VREKAYL7FOL5D6Q7S6VDHDV7GF66D7MGLG4R6OTXRHXAM5MLH3FA, seg_pos={0 0}, upload=112269]: Uploading piece... Dec 10 23:22:19.390092551 (127h53m7.087660858s) [upload=112269]: Next returned nil segment Dec 10 23:22:19.390101177 (127h53m7.087669474s) [upload=112269]: Waiting for error group managing segments... Dec 10 23:22:19.399208490 (127h53m7.096776796s) [noise=true , address=45.91.116.5:4325 , piece_id=VREKAYL7FOL5D6Q7S6VDHDV7GF66D7MGLG4R6OTXRHXAM5MLH3FA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.399514472 (127h53m7.097082778s) [noise=true , address=45.91.116.11:4120 , piece_id=VGO5RNFPIQMUQCOADYBVY4BDHAU7W7DG7CYZPYIFVXWX4RIOJWIQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.400212956 (127h53m7.097781262s) [noise=true , address=45.91.116.12:4259 , piece_id=HCJAMWQHPOPL7KTV25OGFY4WWQ6TEZ4ZPNNC5V7OSBRS25OCKD2Q, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.400525831 (127h53m7.098094137s) [noise=true , address=45.91.116.13:4158 , piece_id=UHS7QZKTMBNF6SLE2EGSMZRI35CRG6H6I6SA2M5FKUFC3JL3YX2Q, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.403193243 (127h53m7.10076155s) [noise=true , address=145.79.227.33:4890 , piece_id=P3PXFVQ6HGZZLQ6PO4LNYT7YYBVB3QMRGTVMR24PYNDKCSPDXWLQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.407584025 (127h53m7.105152341s) [noise=true , address=145.79.227.17:4986 , piece_id=PD7TVILVPBO62NG4PZZRWV25WRDZAAT2W6CLZTLKDX4XSUZE4WVQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.413066777 (127h53m7.110635113s) [noise=true , address=145.79.227.24:4594 , piece_id=YAZHOUIDI4R5OOKOWVHWF6CXMRJTQTZGNAG7RKBXHPVZ3F7M3RLA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.418819682 (127h53m7.116388018s) [noise=true , address=145.79.227.12:4772 , piece_id=7HZPDMV4TPK3DSN42RIGQU7JHIHI43WHODVZAMKQ4AEEJ6QSNB6Q, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.447782774 (127h53m7.14535108s) [noise=true , address=145.79.227.42:4152 , piece_id=3ZTLZDEW3OHVL2CO4FJFLLQ4XEOGJLD76TBLYTAH2PWHO4DNUNYA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.895771245 (127h53m7.593339561s) [noise=true , address=193.149.164.143:4696 , piece_id=NLC3MRZEAXPWIS7VS3ARQPJENDRYIJ5LQAACRNFQEOXI3FU5PSNA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:22:19.896388647 (127h53m7.593956953s) [noise=true , address=64.49.13.139:4473 , piece_id=OAMM5RBJAW4ICUCPG4YZJZSXGVJACZE6OKQ4D65PHVVNIFWWJYTQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: <nil> Dec 10 23:23:19.271658268 (127h54m6.969226574s) [noise=true , address=77.90.31.11:4576 , piece_id=Y6PZNRDF2XLI3YHXR5D6TYD2TTR665L3F4KIHRLSI5YIK4D4HTMA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: write tcp 5.161.89.252:63890->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.271727388 (127h54m6.969295694s) [noise=true , address=77.90.31.10:4536 , piece_id=ZTIQDNF7POKXV7SPJJEY4KYMPBIYPKPCFEJH6ODFL2L5EXZ6DR3Q, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.271858803 (127h54m6.969427119s) [noise=true , address=77.90.31.17:4790 , piece_id=4ZH7QCL6INDT7OF2XOQVYSFZ3IEC5OOCBESWJ5P3AVXQF3JUFQBA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.271946406 (127h54m6.969514713s) [noise=true , address=77.90.31.14:4696 , piece_id=SOVODD7CSFQ3BS5ACB6KY7V6M7KWSFMYBUE4NSAMX6UUTWK36MNQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: rpc: dial tcp 77.90.31.14:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.272089645 (127h54m6.96965795s) [noise=true , address=77.90.31.13:4652 , piece_id=O2PYHR2UO6ZJZ7PZ7EMUEZHGC35QDWHSYMKGSQQB3R77QUZQNMYQ, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: rpc: dial tcp 77.90.31.13:4652: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.272106446 (127h54m6.969674762s) [noise=true , address=77.90.31.12:4601 , piece_id=QR3XV27RH4K5DAF434KZTRRTTSVRAX6POF52IINRWLVB7WKAVXKA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: rpc: dial tcp 77.90.31.12:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.272231689 (127h54m6.969800015s) [noise=true , address=77.90.31.21:4887 , piece_id=7A3D2GSN4MFQZJ5ID4S7PUHXEVGEO3NUH7A7WOQHP3RWW6TBU4IA, seg_pos={0 0}, upload=112269]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: write tcp 5.161.89.252:23166->77.90.31.21:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:19.272257648 (127h54m6.969825955s) [seg_pos={0 0}, upload=112269]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:23:19.286708892 (127h54m6.984277198s) [upload=112269]: Done uploading segments... Dec 10 23:23:19.286729450 (127h54m6.984297766s) [upload=112269]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.016703387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001246032s trace: https://jaeger.dev.storj.io/trace/2637a749f06dbf5 FAIL (2025-12-10 23:23:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001259347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000675294s trace: https://jaeger.dev.storj.io/trace/761fb4d9fefabcd1 FAIL (2025-12-10 23:22:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00071046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 487.284042ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:49:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.296846ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 555.194264ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:48:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.215463ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 492.078697ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:47:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 492.124773ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 494.890488ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:46:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.915505ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 504.573645ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:46:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.590848ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 182.017705ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:45:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 182.029847ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 514.760724ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:45:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.770873ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 500.339296ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:45:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.358712ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 171.927778ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 171.946632ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 906.073482ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:44:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 906.093701ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 268.23996ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:44:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 268.249318ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 310.971136ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:43:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 311.009495ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 172.85065ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:43:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 172.884635ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 724.162785ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 724.185718ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 192.810683ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:42:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 192.831893ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 507.289509ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:42:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.308243ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 192.804811ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:41:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 192.817305ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 489.011799ms trace: https://jaeger.dev.storj.io/trace/7900583b83bc64b7 FAIL (2025-12-10 20:41:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.025045ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 195.355998ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:41:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 195.370544ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 193.505039ms trace: https://jaeger.dev.storj.io/trace/538bb629beb4dbd6 FAIL (2025-12-10 20:41:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 193.518545ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)