Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 476.520155ms FAIL (2025-11-28 22:37:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.540903ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 488.780567ms FAIL (2025-11-28 22:37:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 488.801396ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 472.378565ms FAIL (2025-11-28 22:36:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.396198ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............................X.......] duration: 344.534457ms FAIL (2025-11-28 22:36:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 344.549334ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 509.43544ms FAIL (2025-11-28 22:36:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 509.464114ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 458.18771ms FAIL (2025-11-28 22:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.226893ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 541.825513ms FAIL (2025-11-28 22:35:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 541.850079ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............................X......] duration: 366.538306ms FAIL (2025-11-28 22:35:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 366.680986ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 489.332521ms FAIL (2025-11-28 22:35:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.355865ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 467.038263ms FAIL (2025-11-28 22:34:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.060115ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 392.935624ms FAIL (2025-11-28 22:34:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 392.958157ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............................X.....] duration: 367.941323ms FAIL (2025-11-28 22:34:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 367.961461ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 479.399016ms FAIL (2025-11-28 22:34:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.419805ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 383.474043ms FAIL (2025-11-28 22:34:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 383.49428ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 465.769815ms FAIL (2025-11-28 22:33:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.78782ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 470.230415ms FAIL (2025-11-28 22:33:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.287752ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............................X....] duration: 348.957856ms FAIL (2025-11-28 22:33:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 348.976791ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............................X...] duration: 402.521349ms FAIL (2025-11-28 22:33:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 402.538891ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 523.255889ms FAIL (2025-11-28 22:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 523.29899ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 444.608863ms FAIL (2025-11-28 22:32:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 444.6281ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............................X..] duration: 385.256498ms FAIL (2025-11-28 22:32:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.27876ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 502.145598ms FAIL (2025-11-28 22:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.163532ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 417.354459ms FAIL (2025-11-28 22:32:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 417.378876ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............................X.] duration: 470.209845ms FAIL (2025-11-28 22:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.244009ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 573.640359ms FAIL (2025-11-28 22:32:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.664444ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............................X] duration: 30.000321421s FAIL (2025-11-28 22:31:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000342631s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5.001136247s FAIL (2025-11-28 22:31:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00115427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 351.476531ms FAIL (2025-11-28 22:30:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 351.496068ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 375.115213ms FAIL (2025-11-28 22:30:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 375.136793ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 390.026758ms FAIL (2025-11-28 22:29:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.040744ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 503.905092ms FAIL (2025-11-28 22:29:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 503.922534ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 437.666208ms FAIL (2025-11-28 22:29:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.689701ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 529.569894ms FAIL (2025-11-28 22:29:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 529.599761ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 405.849534ms FAIL (2025-11-28 22:28:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.868731ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 494.42655ms FAIL (2025-11-28 22:28:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 494.461176ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 347.856398ms FAIL (2025-11-28 22:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 347.873901ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 453.01187ms FAIL (2025-11-28 22:27:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 453.035843ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 339.868103ms FAIL (2025-11-28 22:27:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 339.88366ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 422.29561ms FAIL (2025-11-28 22:27:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 422.315757ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 401.64672ms FAIL (2025-11-28 22:26:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 401.657851ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 421.233737ms FAIL (2025-11-28 22:26:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 421.259396ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 462.680039ms FAIL (2025-11-28 22:25:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 462.705178ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 404.166417ms FAIL (2025-11-28 22:25:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 404.183808ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 396.463753ms FAIL (2025-11-28 22:25:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.480545ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 375.19587ms FAIL (2025-11-28 22:24:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 375.225656ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 381.730873ms FAIL (2025-11-28 22:24:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.793531ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 415.039584ms FAIL (2025-11-28 22:23:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 415.060114ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 399.300679ms FAIL (2025-11-28 22:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 399.4774ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 342.907314ms FAIL (2025-11-28 22:23:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 342.919978ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 391.809162ms FAIL (2025-11-28 22:22:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 685337]: Got next segment. Inline: false Nov 28 22:07:45.755017846 (778h54m12.455962761s) [upload=685337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:45.755023667 (778h54m12.455968582s) [upload=685337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:45.778708895 (778h54m12.47965379s) [seg_pos={0 0}, upload=685337]: Begin upload segment... Nov 28 22:07:45.778745854 (778h54m12.479690749s) [seg_pos={0 0}, upload=685337]: Done begin upload segment. Nov 28 22:07:45.778765340 (778h54m12.479710225s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.778775700 (778h54m12.479720595s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779062417 (778h54m12.480007303s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779307377 (778h54m12.480252272s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779575400 (778h54m12.480520295s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.779823907 (778h54m12.480768802s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780073154 (778h54m12.481018039s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780325457 (778h54m12.481270352s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780580225 (778h54m12.48152512s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.780820516 (778h54m12.481765401s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781491926 (778h54m12.482436821s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.781855017 (778h54m12.482799912s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782200475 (778h54m12.48314537s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782505077 (778h54m12.483449972s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.782688932 (778h54m12.483633827s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783017358 (778h54m12.483962253s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783223365 (778h54m12.48416826s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.783599079 (778h54m12.484543964s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Uploading piece... Nov 28 22:07:45.846053485 (778h54m12.54699838s) [upload=685337]: Next returned nil segment Nov 28 22:07:45.846070156 (778h54m12.547015141s) [upload=685337]: Waiting for error group managing segments... Nov 28 22:07:45.854930911 (778h54m12.555875815s) [noise=true , address=145.79.227.18:4038 , piece_id=ZZLRO4YRCWTEDUWH6NOO5MXGK4XYOB7HIQZ6OM7N2H55RABW62JA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855091713 (778h54m12.556036617s) [noise=true , address=77.90.31.17:4778 , piece_id=M66OFE6QCOKUL6M4T6WSEMVKG7BQ3C3DTGNHOB6LDMOUYWJOP3PA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.855902954 (778h54m12.556847859s) [noise=true , address=145.79.227.33:4878 , piece_id=HNWAJWLLCSRYTRSHBTEDOJWPQEYWQREX2DX3KMIDIATTE6PHBWZQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857845339 (778h54m12.558790243s) [noise=true , address=77.90.31.15:4719 , piece_id=5XS752NHATLM2JJWZDQFYPQVIBIBCLMMIY636SL4W2IBD2F6P3GQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.857955355 (778h54m12.55890026s) [noise=true , address=145.79.227.44:4206 , piece_id=2XPEXIA3AJIEUSNJLQNJP7WQZKOYO3TC2F6ESOAKOE5WQT2XEZWQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.858253525 (778h54m12.559198429s) [noise=true , address=77.90.31.18:4810 , piece_id=VT5BQGHINSMIRB66KCPQYHBUNIHHWTR6HV25Y42S4KUUK6TUPP3Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864341055 (778h54m12.565285971s) [noise=true , address=77.90.31.21:4887 , piece_id=SECEKOXHHUQX2SOESFJL3VHDI676CZMQWDITLFXHBGU6IDGRT4BQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.864818060 (778h54m12.565762965s) [noise=true , address=145.79.227.12:4796 , piece_id=PEEPORM2VQQOC62VUMCGA4KIOVOV2CGB4L54X4RXNIUPUADQXJMA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.868107551 (778h54m12.569052467s) [noise=true , address=145.79.227.9:4389 , piece_id=XEMIPG7O4VK77K5R3R3HH4NE5AKATF2NN2L6QCI47P4AEHDLEV5A, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.875789043 (778h54m12.576733959s) [noise=true , address=77.90.31.12:4617 , piece_id=BAKZHAB47DY5EWA5W6GQK3XZPKCLKN6OK4RUGMM2RL2KMV5SJP6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:45.882872613 (778h54m12.583817529s) [noise=true , address=145.79.227.16:4938 , piece_id=KYSYENOVG65C4MVJ5GFFJCEAYY6QPGNUMZFKLBYPT6A255MIHSSQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:46.354270809 (778h54m13.055215704s) [noise=true , address=64.49.13.170:4722 , piece_id=JQZIRS4W4NYMC4S3BKV7II53RWTWBCFGJD7GURBZVIOJ46NEEIRA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:07:47.317446642 (778h54m14.018391537s) [noise=true , address=64.49.13.169:4710 , piece_id=RHYEQNIKAYKM3TCHHXXEWPZT3FDQ4JIJKILWK43G57MIZ36E4CBA, seg_pos={0 0}, upload=685337]: Done uploading piece. err: <nil> Nov 28 22:08:45.741611849 (778h55m12.442556744s) [noise=true , address=45.91.116.14:4220 , piece_id=DPOFFPVE6OREBRPYCGL5IG3CHRJJSCTAG55CQKIW6U5CCGHPEHHQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, address:45.91.116.14:4220): protocol: rpc: dial tcp 45.91.116.14:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741625925 (778h55m12.44257082s) [noise=true , address=45.91.116.7:4033 , piece_id=JXOQUCARCQ47W3UIQBERGIYPYBG2NZEWZ7OE2BC5NVIBPAPSCE2Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: write tcp 5.161.89.252:39072->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741709111 (778h55m12.442654006s) [noise=true , address=45.91.116.7:4025 , piece_id=QVQD6ZTBWZCZ4THMPRUP5IA6RCM7KG33LU5QTPVHEB6WPU2AKO5Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 5.161.89.252:12040->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741732926 (778h55m12.442677821s) [noise=true , address=45.91.116.13:4159 , piece_id=R3Z55SUVTEULT2XGGOOGKJLDL4QIXT4R3HODMSMAH5OSBFISUKVQ, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741776548 (778h55m12.442721432s) [noise=true , address=45.91.116.11:4122 , piece_id=BOSR3QZSILW2GW4P3DRLQCG332ODFY44DATMHO3MESSEGESJXW6Q, seg_pos={0 0}, upload=685337]: Done uploading piece. err: ecclient: upload failed (node:1Kg43gDX5Tybqk93Xg9F1rVzGjbDGJYrrkDUUK4Mepdu2iGYJm, address:45.91.116.11:4122): protocol: read tcp 5.161.89.252:8108->45.91.116.11:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:45.741795153 (778h55m12.442740048s) [seg_pos={0 0}, upload=685337]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 28 22:08:45.761066499 (778h55m12.462011383s) [upload=685337]: Done uploading segments... Nov 28 22:08:45.761081336 (778h55m12.462026221s) [upload=685337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 391.830862ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)