| *** betherly has joined #openstack-infra | 00:02 | |
| *** Qiming has joined #openstack-infra | 00:04 | |
| *** mattw4 has quit IRC | 00:05 | |
| *** betherly has quit IRC | 00:07 | |
| *** Qiming has quit IRC | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:10 | |
| *** Qiming has joined #openstack-infra | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| *** Qiming has quit IRC | 00:22 | |
| *** Qiming has joined #openstack-infra | 00:23 | |
| *** betherly has joined #openstack-infra | 00:23 | |
| *** slaweq has quit IRC | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** betherly has quit IRC | 00:28 | |
| *** auristor has quit IRC | 00:28 | |
| *** gyee has quit IRC | 00:30 | |
| *** auristor has joined #openstack-infra | 00:34 | |
| *** wolverineav has quit IRC | 00:42 | |
| *** markvoelker has joined #openstack-infra | 00:43 | |
| *** betherly has joined #openstack-infra | 00:44 | |
| *** wolverineav has joined #openstack-infra | 00:46 | |
| *** betherly has quit IRC | 00:49 | |
| *** wolverineav has quit IRC | 00:51 | |
| *** whoami-rajat has joined #openstack-infra | 00:51 | |
| *** wolverineav has joined #openstack-infra | 00:53 | |
| *** tkajinam_ has joined #openstack-infra | 00:58 | |
| *** tkajinam has quit IRC | 01:01 | |
| *** betherly has joined #openstack-infra | 01:05 | |
| *** betherly has quit IRC | 01:10 | |
| *** armax has joined #openstack-infra | 01:11 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** rlandy has quit IRC | 01:16 | |
| *** markvoelker has quit IRC | 01:21 | |
| *** jamesmcarthur has joined #openstack-infra | 01:23 | |
| *** slaweq has quit IRC | 01:24 | |
| *** agopi has joined #openstack-infra | 01:24 | |
| *** betherly has joined #openstack-infra | 01:26 | |
| *** rascasoft has joined #openstack-infra | 01:29 | |
| *** betherly has quit IRC | 01:31 | |
| *** wolverineav has quit IRC | 01:31 | |
| *** wolverineav has joined #openstack-infra | 01:32 | |
| *** zbr has quit IRC | 01:34 | |
| *** wolverineav has quit IRC | 01:37 | |
| *** rascasoft has quit IRC | 01:38 | |
| *** betherly has joined #openstack-infra | 02:08 | |
| *** betherly has quit IRC | 02:12 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** slaweq has joined #openstack-infra | 02:26 | |
| *** betherly has joined #openstack-infra | 02:29 | |
| *** ricolin_ has joined #openstack-infra | 02:31 | |
| *** betherly has quit IRC | 02:34 | |
| *** slaweq has quit IRC | 02:39 | |
| *** betherly has joined #openstack-infra | 02:50 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** betherly has quit IRC | 02:54 | |
| *** yamamoto has quit IRC | 02:55 | |
| *** yamamoto has joined #openstack-infra | 02:55 | |
| *** yamamoto has quit IRC | 02:56 | |
| *** psachin has joined #openstack-infra | 03:03 | |
| *** yamamoto has joined #openstack-infra | 03:08 | |
| *** yamamoto has quit IRC | 03:09 | |
| *** yamamoto has joined #openstack-infra | 03:10 | |
| *** betherly has joined #openstack-infra | 03:11 | |
| *** slaweq has joined #openstack-infra | 03:12 | |
| *** yamamoto has quit IRC | 03:15 | |
| *** betherly has quit IRC | 03:15 | |
| *** apetrich has quit IRC | 03:16 | |
| *** roman_g has quit IRC | 03:20 | |
| *** rascasoft has joined #openstack-infra | 03:23 | |
| *** slaweq has quit IRC | 03:24 | |
| *** diablo_rojo has quit IRC | 03:26 | |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** ramishra has joined #openstack-infra | 03:27 | |
| *** betherly has joined #openstack-infra | 03:31 | |
| jroll | corvus: apologies for missing today's meeting, I'll be at the rest for the foreseeable future. at HQ doing team building things this week | 03:31 |
|---|---|---|
| jroll | corvus: I will start conversation around openstack/ namespace this week :) | 03:32 |
| *** rascasoft has quit IRC | 03:32 | |
| *** eernst has quit IRC | 03:35 | |
| *** betherly has quit IRC | 03:36 | |
| *** armax has quit IRC | 03:41 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** ricolin has joined #openstack-infra | 04:13 | |
| *** ricolin_ has quit IRC | 04:13 | |
| *** udesale has joined #openstack-infra | 04:17 | |
| *** rfolco|ruck has quit IRC | 04:17 | |
| *** ykarel|away has joined #openstack-infra | 04:22 | |
| *** slaweq has quit IRC | 04:24 | |
| *** rfolco has joined #openstack-infra | 04:24 | |
| *** rh-jelabarre has quit IRC | 04:29 | |
| *** rfolco has quit IRC | 04:30 | |
| *** rfolco has joined #openstack-infra | 04:30 | |
| *** betherly has joined #openstack-infra | 04:44 | |
| *** betherly has quit IRC | 04:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 05:04 |
| *** janki has joined #openstack-infra | 05:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Implement zookeeper-auth https://review.openstack.org/619156 | 05:04 |
| *** betherly has joined #openstack-infra | 05:05 | |
| *** betherly has quit IRC | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: add job's tags to the rpc job_list method https://review.openstack.org/633653 | 05:12 |
| *** wolverineav has joined #openstack-infra | 05:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add tags to jobs list https://review.openstack.org/633654 | 05:14 |
| *** udesale has quit IRC | 05:16 | |
| *** udesale has joined #openstack-infra | 05:17 | |
| *** jaosorior has joined #openstack-infra | 05:22 | |
| *** slaweq has quit IRC | 05:24 | |
| *** markvoelker has joined #openstack-infra | 05:47 | |
| *** udesale has quit IRC | 05:51 | |
| *** hwoarang has quit IRC | 05:53 | |
| *** betherly has joined #openstack-infra | 05:57 | |
| *** hwoarang has joined #openstack-infra | 06:00 | |
| *** betherly has quit IRC | 06:01 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** udesale has joined #openstack-infra | 06:11 | |
| *** ykarel|away is now known as ykarel | 06:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Unmount internal mounts on finalise errors https://review.openstack.org/642953 | 06:17 |
| *** pcaruana has joined #openstack-infra | 06:19 | |
| ianw | tobiash: i can't exactly replicate your missing return code issue. But could you try with ^? That should fix leaked mounts, and maybe the problem ... | 06:19 |
| tobiash | ianw: thanks will try :) | 06:19 |
| ianw | i think that change is correct either way, but if not helping maybe has to do with the changes to calling dib recently in nodepool for the timeout | 06:22 |
| tobiash | ianw: my test was not running in the nodepool context but on a fresh build instance | 06:23 |
| ianw | tobiash: hrm, running under nodepool-builder thought? sorry what i mean is the changes made there to handle image build timeouts | 06:24 |
| *** slaweq has quit IRC | 06:24 | |
| *** slaweq has joined #openstack-infra | 06:26 | |
| tobiash | I'm not running this validation in nodepool yet, but manually. But I plan to run this in nodepool. | 06:26 |
| ianw | ahh, ok | 06:28 |
| ianw | yes, i imagine that change will help anyway. because you don't want to leak mounts if/when the build fails | 06:28 |
| ianw | i really hope it's just a typo-level bug, and not because we've baked in some other weird assumptions about mounts. it took me quite a while to pull it apart even that far to grok it :/ | 06:29 |
| *** dayou has quit IRC | 06:30 | |
| *** slaweq has quit IRC | 06:36 | |
| *** yolanda has quit IRC | 06:36 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test zuul test case sharding https://review.openstack.org/642956 | 06:37 |
| *** e0ne has joined #openstack-infra | 06:40 | |
| *** masayukig[m]2 is now known as masayukig[m] | 06:45 | |
| *** masayukig[m] is now known as masayuki4 | 06:45 | |
| *** kjackal has joined #openstack-infra | 06:45 | |
| *** dayou has joined #openstack-infra | 06:47 | |
| *** masayuki4 is now known as masayuki6 | 06:48 | |
| *** masayuki6 is now known as masayukig[m] | 06:48 | |
| *** masayukig[m] is now known as masayukig | 06:48 | |
| *** diablo_rojo has joined #openstack-infra | 06:52 | |
| *** wolverineav has quit IRC | 06:53 | |
| *** kjackal has quit IRC | 07:00 | |
| *** hwoarang has quit IRC | 07:07 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| *** hwoarang has joined #openstack-infra | 07:08 | |
| tobiash | ianw: sorry, I think I found my problem. And it's not diskimage-builder :-/ | 07:08 |
| *** yolanda has joined #openstack-infra | 07:09 | |
| tobiash | ianw: I piped the diskimage-builder output through tee without using pipefail... | 07:09 |
| * tobiash hides | 07:09 | |
| tobiash | ianw: but at least the mount leak seems to be fixed with your change ;) | 07:10 |
| *** slaweq has joined #openstack-infra | 07:12 | |
| *** kjackal has joined #openstack-infra | 07:13 | |
| *** tonyb is now known as tonyb_gone | 07:20 | |
| *** tonyb_gone is now known as tonyb | 07:21 | |
| *** dpawlik has joined #openstack-infra | 07:23 | |
| *** slaweq has quit IRC | 07:24 | |
| *** gbutnaru has quit IRC | 07:29 | |
| *** e0ne has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-infra | 07:35 | |
| *** rascasoft has joined #openstack-infra | 07:36 | |
| *** apetrich has joined #openstack-infra | 07:50 | |
| *** e0ne has joined #openstack-infra | 07:51 | |
| *** wolverineav has joined #openstack-infra | 07:51 | |
| *** dpawlik has quit IRC | 07:54 | |
| *** wolverineav has quit IRC | 07:56 | |
| *** sshnaidm|afk has quit IRC | 07:59 | |
| *** lennyb has joined #openstack-infra | 08:05 | |
| *** dpawlik has joined #openstack-infra | 08:06 | |
| *** ginopc has joined #openstack-infra | 08:08 | |
| *** rcernin has quit IRC | 08:09 | |
| *** rpittau|afk is now known as rpittau | 08:09 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: k8s: Fix delete_namespace to use kwargs properly https://review.openstack.org/642964 | 08:12 |
| *** yamamoto has joined #openstack-infra | 08:19 | |
| *** electrofelix has joined #openstack-infra | 08:19 | |
| ianw | tobiash: well, i'll call that progress :) | 08:20 |
| *** ccamacho has joined #openstack-infra | 08:23 | |
| *** yamamoto has quit IRC | 08:23 | |
| *** pgaxatte has joined #openstack-infra | 08:25 | |
| *** helenafm has joined #openstack-infra | 08:30 | |
| *** e0ne has quit IRC | 08:36 | |
| *** hwoarang has quit IRC | 08:40 | |
| *** hwoarang has joined #openstack-infra | 08:42 | |
| *** roman_g has joined #openstack-infra | 08:43 | |
| tobiash | ianw: I just verified that it fixes the mount leaks :) | 08:45 |
| tobiash | ianw: so that fix is really required so we don't kill all the nodepool builders when the size validation fails :) | 08:46 |
| *** iurygregory has joined #openstack-infra | 08:46 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** xek has joined #openstack-infra | 08:50 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| *** zbr has joined #openstack-infra | 09:05 | |
| *** dtantsur|afk is now known as dtantsur | 09:14 | |
| *** kukacz has quit IRC | 09:15 | |
| *** kukacz has joined #openstack-infra | 09:17 | |
| *** panda|rover|off is now known as panda|rover | 09:17 | |
| *** tkajinam__ has joined #openstack-infra | 09:22 | |
| *** FlorianFa has joined #openstack-infra | 09:23 | |
| *** ruffian_sheep has joined #openstack-infra | 09:23 | |
| *** hwoarang has quit IRC | 09:23 | |
| *** tkajinam_ has quit IRC | 09:25 | |
| *** yamamoto has joined #openstack-infra | 09:27 | |
| ruffian_sheep | Does anyone know what the job the ci should do? | 09:29 |
| *** kjackal_v2 has joined #openstack-infra | 09:30 | |
| *** kjackal_v2 has quit IRC | 09:31 | |
| *** kjackal has quit IRC | 09:31 | |
| *** kjackal has joined #openstack-infra | 09:32 | |
| ruffian_sheep | fungi:Do u know that ?Now I can set some review to show the result of my job. | 09:32 |
| *** hwoarang has joined #openstack-infra | 09:32 | |
| *** rfolco is now known as rfolco|ruck | 09:32 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 09:32 |
| *** derekh has joined #openstack-infra | 09:35 | |
| *** sshnaidm|afk has joined #openstack-infra | 09:36 | |
| *** Emine has joined #openstack-infra | 09:53 | |
| *** sshnaidm|afk has quit IRC | 09:56 | |
| frickler | corvus: still got a bounce from zp01.opendev.org this morning, just as a reminder ;) | 09:58 |
| frickler | ruffian_sheep: are you talking about a job in our CI or a 3rd party CI? please explain your issue in a bit more detail | 10:00 |
| *** diablo_rojo has quit IRC | 10:11 | |
| *** shardy has joined #openstack-infra | 10:18 | |
| *** hwoarang has quit IRC | 10:28 | |
| *** hwoarang has joined #openstack-infra | 10:29 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** luizbag has joined #openstack-infra | 10:39 | |
| *** dave-mccowan has joined #openstack-infra | 10:40 | |
| *** priteau has joined #openstack-infra | 10:45 | |
| *** dave-mccowan has quit IRC | 10:45 | |
| *** verdurin has joined #openstack-infra | 11:01 | |
| *** rossella_s has quit IRC | 11:06 | |
| *** yamamoto has joined #openstack-infra | 11:06 | |
| *** rossella_s has joined #openstack-infra | 11:08 | |
| *** yamamoto has quit IRC | 11:13 | |
| *** tosky has joined #openstack-infra | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:18 | |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Displayed tags should link to search view for the tag https://review.openstack.org/642262 | 11:22 |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** mriedem has joined #openstack-infra | 11:26 | |
| *** sshnaidm|afk has joined #openstack-infra | 11:28 | |
| *** ruffian_sheep has quit IRC | 11:33 | |
| *** e0ne has joined #openstack-infra | 11:36 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for neutron API did not start bug 1819897 https://review.openstack.org/643019 | 11:42 |
| openstack | bug 1819897 in neutron "Intermittent "Neutron did not start" failures in the gate" [Medium,Confirmed] https://launchpad.net/bugs/1819897 | 11:42 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for our .scss files https://review.openstack.org/643025 | 11:52 |
| *** rh-jelabarre has joined #openstack-infra | 11:53 | |
| frickler | mriedem: ^^ I think your query is too broad, seems there are lots of non-gate failures like http://logs.openstack.org/34/642834/1/check/tempest-full/f8a254a/controller/logs/screen-q-svc.txt.gz#_Mar_12_18_16_06_388244 rather than timeouts | 11:53 |
| *** rh-jelabarre has quit IRC | 11:53 | |
| *** rh-jelabarre has joined #openstack-infra | 11:53 | |
| *** luizbag_ has joined #openstack-infra | 11:54 | |
| *** luizbag has quit IRC | 11:55 | |
| mriedem | frickler: there were no errors in the neutron API logs from the one I linked into the bug report | 11:55 |
| *** hwoarang has quit IRC | 11:56 | |
| frickler | mriedem: yes, there probably is a valid issue there, I'm just noting that for most hits of your query that I checked (like the first five), the failure looked different | 11:57 |
| *** sshnaidm|afk is now known as sshnaidm | 11:57 | |
| *** hwoarang has joined #openstack-infra | 11:58 | |
| frickler | mriedem: so I think we need to discriminate between "q-svc start took > 60sec" and "q-svc failed to start completely" | 11:59 |
| mriedem | i'm not totally sure if that's possible in the logstash query | 11:59 |
| *** jpena|lunch is now known as jpena | 12:00 | |
| frickler | yes, might need better checking in devstack | 12:02 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for neutron API did not start bug 1819897 https://review.openstack.org/643019 | 12:07 |
| openstack | bug 1819897 in neutron "Intermittent "Neutron did not start" failures in the gate" [Medium,Confirmed] https://launchpad.net/bugs/1819897 | 12:07 |
| *** markvoelker has quit IRC | 12:12 | |
| *** markvoelker has joined #openstack-infra | 12:13 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for HTML templates https://review.openstack.org/643032 | 12:17 |
| *** rlandy has joined #openstack-infra | 12:21 | |
| *** hwoarang has quit IRC | 12:24 | |
| *** hwoarang has joined #openstack-infra | 12:28 | |
| *** wolverineav has joined #openstack-infra | 12:29 | |
| *** panda|rover is now known as panda|rover|eat | 12:30 | |
| *** wolverineav has quit IRC | 12:34 | |
| *** janki has quit IRC | 12:34 | |
| *** lseki has joined #openstack-infra | 12:44 | |
| *** lseki has quit IRC | 12:45 | |
| *** lseki has joined #openstack-infra | 12:45 | |
| *** jamesmcarthur has joined #openstack-infra | 12:47 | |
| *** janki has joined #openstack-infra | 12:58 | |
| *** panda|rover|eat is now known as panda|rover | 13:00 | |
| *** e0ne has quit IRC | 13:05 | |
| *** gfidente has joined #openstack-infra | 13:14 | |
| *** udesale has quit IRC | 13:16 | |
| *** udesale has joined #openstack-infra | 13:17 | |
| *** tosky has quit IRC | 13:22 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for HTML templates https://review.openstack.org/643032 | 13:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Use the latest active LTS version (10) of Node.js for testing https://review.openstack.org/643046 | 13:26 |
| *** priteau has quit IRC | 13:27 | |
| *** eharney has joined #openstack-infra | 13:37 | |
| *** jamesmcarthur has quit IRC | 13:41 | |
| *** yamamoto has quit IRC | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** yamamoto has quit IRC | 13:54 | |
| *** Emine has quit IRC | 13:59 | |
| *** jamesmcarthur has joined #openstack-infra | 14:00 | |
| *** jamesmcarthur has quit IRC | 14:00 | |
| *** hwoarang has quit IRC | 14:01 | |
| *** jamesmcarthur has joined #openstack-infra | 14:01 | |
| *** luizbag has joined #openstack-infra | 14:03 | |
| *** luizbag_ has quit IRC | 14:05 | |
| *** hwoarang has joined #openstack-infra | 14:08 | |
| *** Emine has joined #openstack-infra | 14:08 | |
| *** irclogbot_1 has quit IRC | 14:09 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Update to a newer version (8) of Node.js for testing https://review.openstack.org/643046 | 14:10 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for HTML templates https://review.openstack.org/643032 | 14:10 |
| *** e0ne has joined #openstack-infra | 14:10 | |
| *** irclogbot_1 has joined #openstack-infra | 14:12 | |
| *** yamamoto has joined #openstack-infra | 14:16 | |
| *** hwoarang has quit IRC | 14:17 | |
| *** hwoarang has joined #openstack-infra | 14:19 | |
| *** bnemec has quit IRC | 14:22 | |
| *** eharney has quit IRC | 14:23 | |
| *** Emine has quit IRC | 14:24 | |
| *** Emine has joined #openstack-infra | 14:24 | |
| *** irclogbot_1 has quit IRC | 14:25 | |
| *** bnemec has joined #openstack-infra | 14:27 | |
| *** irclogbot_1 has joined #openstack-infra | 14:27 | |
| *** eharney has joined #openstack-infra | 14:28 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Update to a newer version (8) of Node.js for testing https://review.openstack.org/643046 | 14:30 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for HTML templates https://review.openstack.org/643032 | 14:30 |
| *** tobasco is now known as tobias-urdin | 14:32 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Update to a newer version (8) of Node.js for testing https://review.openstack.org/643046 | 14:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a linter for HTML templates https://review.openstack.org/643032 | 14:34 |
| *** luizbag_ has joined #openstack-infra | 14:34 | |
| *** luizbag has quit IRC | 14:35 | |
| *** jtomasek has quit IRC | 14:38 | |
| EmilienM | mordred, clarkb : heads-up - the docker registry v2 (served on 8082) seems to work fine | 14:39 |
| EmilienM | we are transitioning | 14:39 |
| EmilienM | http://mirror.ord.rax.openstack.org:8082 "GET /v2/tripleomaster/centos-binary-aodh-api/tags/list HTTP/1.1" 200 None | 14:39 |
| EmilienM | http://logs.openstack.org/08/642808/3/check/tripleo-ci-centos-7-scenario001-standalone/ce36cc8/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz#_2019-03-13_12_37_10_132 | 14:40 |
| *** ramishra has quit IRC | 14:41 | |
| *** agopi has quit IRC | 14:42 | |
| *** agopi has joined #openstack-infra | 14:42 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for DestinationHypervisorTooOld bug 1819910 https://review.openstack.org/643064 | 14:52 |
| openstack | bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,In progress] https://launchpad.net/bugs/1819910 - Assigned to Matt Riedemann (mriedem) | 14:52 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Add missing doc for 'execute_time' in MQTT schema https://review.openstack.org/643069 | 15:03 |
| *** psachin has quit IRC | 15:07 | |
| *** armax has joined #openstack-infra | 15:08 | |
| *** rfolco|ruck is now known as rfolco | 15:10 | |
| *** mriedem has quit IRC | 15:10 | |
| *** mriedem has joined #openstack-infra | 15:11 | |
| *** lpetrut has joined #openstack-infra | 15:14 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for DestinationHypervisorTooOld bug 1819910 https://review.openstack.org/643064 | 15:17 |
| openstack | bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,In progress] https://launchpad.net/bugs/1819910 - Assigned to Matt Riedemann (mriedem) | 15:17 |
| clarkb | EmilienM: good to hear | 15:21 |
| *** martinkennelly has joined #openstack-infra | 15:21 | |
| fungi | EmilienM: that's excellent news! it's possible tripleo is the last holdout for the old v1 proxy, in which case we might be able to remove it soon | 15:21 |
| clarkb | fungi: we should double check with kolla and loci as well, but ya I would expect that docker pulls aversion to v1 would've forced them to move laready | 15:22 |
| EmilienM | please let us a bit of time to make sure it's all stable | 15:22 |
| EmilienM | then +1 for retiring v1 | 15:22 |
| frickler | infra-root: looking into slow paste.o.o responses, it seems I found a bunch of spam like http://paste.openstack.org/show/636274/ , matching accesses in the server log with a "curb" client. not sure how to handle these | 15:22 |
| clarkb | corvus: mordred I've got a call in a few minutes but after that I intend to start the gitea fixup playbook. I'll probably run it in screen and tee the output into a log file | 15:22 |
| *** dpawlik has quit IRC | 15:22 | |
| frickler | also not sure whether or not those might be related to slow responses | 15:23 |
| clarkb | frickler: paste does hae a captcha thing, but I wonder if they are bypassing that. Also possible that it isn't triggering for those posts | 15:23 |
| clarkb | (fwiw the original authors of that paste software gave up running their public instance for this reason :( ) | 15:23 |
| clarkb | if the spammers originate from a small number of IP addrs we could block them via the firewall but I dobut that do | 15:25 |
| fungi | EmilienM: yes, i'm not saying we schedule removing the old service, just that you're providing hope we might eventually be able to do so ;) | 15:25 |
| fungi | frickler: we do at least have http://paste.openstack.org/robots.txt telling reputable search engines not to index anything there, so spammers have minimal incentive to actually put things in it | 15:26 |
| frickler | clarkb: I'm not seeing any captcha when submitting a paste | 15:27 |
| fungi | this reduced the volume of spam we were seeing there originally by many orders of magnitude | 15:27 |
| clarkb | frickler: it is conditional to the paste content | 15:27 |
| fungi | frickler: it's only (or at least mostly) for content with urls in it | 15:27 |
| clarkb | frickler: I don't know how it determines when to trigger. I usually hit it when pasting code | 15:27 |
| fungi | and also it's only done when posting via the form but i've been able to bypass it with a pastebinit client with no trouble in scripted operations | 15:28 |
| fungi | but also, solutions to simple captchas like that are easily automated around so even if it weren't trivially bypassed it wouldn't make a lot of difference | 15:29 |
| frickler | hmm, ignoring the spam, it also seems we are not logging the time it took to serve a request, which makes debugging slow responses difficult. I'll take a look at that tomorrow, unless someone has a solution handy already | 15:30 |
| *** hwoarang has quit IRC | 15:32 | |
| *** hwoarang has joined #openstack-infra | 15:33 | |
| *** ginopc has quit IRC | 15:33 | |
| *** ginopc has joined #openstack-infra | 15:35 | |
| *** irclogbot_1 has quit IRC | 15:36 | |
| *** ykarel is now known as ykarel|away | 15:37 | |
| *** irclogbot_1 has joined #openstack-infra | 15:39 | |
| *** chandankumar is now known as raukadah | 15:47 | |
| *** eharney has quit IRC | 15:49 | |
| *** IvensZambrano has joined #openstack-infra | 15:49 | |
| *** irclogbot_1 has quit IRC | 15:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add cgit-alias for zuul-preview https://review.openstack.org/643083 | 15:49 |
| pabelanger | clarkb: mordred: corvus: fungi: ^follow up from #zuul, I think we want to add zuul-preview to our cgit-alias for zuul, right? | 15:50 |
| *** irclogbot_1 has joined #openstack-infra | 15:50 | |
| *** irclogbot_1 has quit IRC | 15:52 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/openstack-zuul-jobs master: Disable releasenote sync to Zanata temporary https://review.openstack.org/643085 | 15:54 |
| *** irclogbot_1 has joined #openstack-infra | 15:56 | |
| fungi | good call | 15:58 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Enhance adding tags on story details page https://review.openstack.org/642643 | 16:00 |
| *** diablo_rojo has joined #openstack-infra | 16:01 | |
| clarkb | frickler: I am not aware of a solution to that already | 16:01 |
| *** electrofelix has quit IRC | 16:02 | |
| *** mattw4 has joined #openstack-infra | 16:02 | |
| *** mattw4 has quit IRC | 16:03 | |
| clarkb | fungi: what is the schedule/plan for xenial -> bionic moves? are there changes that need review and or things to monitor? | 16:05 |
| clarkb | (I'm going to start the gitea fixup shortly but that is expected to take a while so can help with job updates too) | 16:05 |
| clarkb | corvus: mordred `ansible-playbook -f 10 /opt/system-config/playbooks/sync-gitea-projects.yaml 2>&1 | tee /home/clarkb/gitea_fixup.log` that look right to you? | 16:09 |
| corvus | clarkb: yep | 16:11 |
| clarkb | ok I'll start that in a root screen | 16:11 |
| clarkb | it is running now | 16:12 |
| fungi | clarkb: the plan is i'm trying to coordinate with gmann's legacy job transition to bionic, but the base job nodeset change we'll want to approve in concert with his work being ready is 641897 | 16:12 |
| fungi | it seems like getting his changes merged just before or at roughly the same time as the base job change will reduce confusion, since it should keep legacy jobs running on xenial for current stable branches at least | 16:13 |
| fungi | he was suggesting he expected to be ready by the end of his day today (pretty sure he's in the americas as of recently) | 16:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Add cgit-alias for zuul-preview https://review.openstack.org/643083 | 16:14 |
| gmann | fungi: yeah, most probably by 4 PM CST that's what i am setting the target. currently working on few nova fix and frickler on designate one. | 16:15 |
| *** jtomasek has joined #openstack-infra | 16:16 | |
| *** trown is now known as trown|lunch | 16:18 | |
| clarkb | infra-root https://opendev.org/openstack-dev/devstack click the issues tab :) | 16:19 |
| *** mattw4 has joined #openstack-infra | 16:19 | |
| corvus | clarkb: \o/ | 16:19 |
| *** Emine has quit IRC | 16:20 | |
| clarkb | https://opendev.org/openstack-dev/devstack/commit/96269d8e758e168fb857fca6e96cbd9a50628ac0 has a clickable bug number that will take you straight to the bug too | 16:20 |
| corvus | excellent! | 16:20 |
| clarkb | need to verify a grouped project in launchpad also works but I'll probably wait for it to get further along first | 16:21 |
| clarkb | (since I'm not sure there are any early in the list) | 16:22 |
| *** priteau has joined #openstack-infra | 16:24 | |
| *** irclogbot_1 has quit IRC | 16:24 | |
| *** irclogbot_1 has joined #openstack-infra | 16:25 | |
| *** gyee has joined #openstack-infra | 16:25 | |
| clarkb | it seems to have stopped, does it batch process the entire list of projects for each org then continue once done? I think that may be what is happening | 16:26 |
| *** pgaxatte has quit IRC | 16:26 | |
| clarkb | and ya it is moving against so each org must have a batch processing happen upfront then it goes repo by repo | 16:29 |
| *** wolverineav has joined #openstack-infra | 16:36 | |
| *** wolverineav has quit IRC | 16:39 | |
| *** wolverineav has joined #openstack-infra | 16:39 | |
| *** helenafm has quit IRC | 16:40 | |
| *** ykarel|away is now known as ykarel | 16:41 | |
| *** FlorianFa has quit IRC | 16:41 | |
| *** udesale has quit IRC | 16:42 | |
| *** dave-mccowan has joined #openstack-infra | 16:42 | |
| *** lpetrut has quit IRC | 16:44 | |
| *** kgiusti has joined #openstack-infra | 16:48 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Make task ordering in Story view selectable https://review.openstack.org/641930 | 16:50 |
| *** ykarel is now known as ykarel|away | 16:51 | |
| *** FlorianFa has joined #openstack-infra | 16:54 | |
| *** e0ne has quit IRC | 17:01 | |
| clarkb | as an fyi I claered out the old zuul status backup files on zuul01 | 17:02 |
| clarkb | the reorg there has them writing to different file names so we stopped pruning the old filenames. This should be the last piece of cleanup necessary around that | 17:02 |
| fungi | thanks! | 17:03 |
| *** lpetrut has joined #openstack-infra | 17:04 | |
| *** wolverineav has quit IRC | 17:04 | |
| *** wolverineav has joined #openstack-infra | 17:06 | |
| *** trown|lunch is now known as trown | 17:07 | |
| *** wolverineav has quit IRC | 17:11 | |
| *** wolverineav has joined #openstack-infra | 17:11 | |
| fungi | frickler: one thing i notice about the trove instance for paste.o.o is it's filling up... now at 3.8 of 5 gb allocated for the disk | 17:14 |
| fungi | it's also completely using the 2 gb of ram allocated for it according to rackspace's dashboard | 17:15 |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| fungi | cpu utilization and load average seem nominal though, so we could probably get away with just resizing the instance for more ram and more disk | 17:16 |
| *** gfidente has quit IRC | 17:16 | |
| *** rpittau is now known as rpittau|afk | 17:16 | |
| fungi | infra-root: any objection to me doubling the paste.o.o trove instance to 10gb disk and 4gb ram? | 17:17 |
| *** gfidente has joined #openstack-infra | 17:17 | |
| clarkb | not from me | 17:17 |
| fungi | i'll be sure to snapshot it immediately prior just in case something with the resize goes sideways | 17:19 |
| *** gyee has quit IRC | 17:20 | |
| *** gyee has joined #openstack-infra | 17:20 | |
| frickler | infra-root: doing "sudo nodepool list" gives "openstack.exceptions.ConfigException: Cloud rax was not found.". this is my usual command to locate held nodes and did work until at least a couple of days ago | 17:21 |
| frickler | ehm ... on nl01.o.o that is | 17:21 |
| fungi | frickler: `sudo -u nodepool nodepool list` maybe? | 17:22 |
| fungi | er, needs -H too | 17:22 |
| clarkb | the cloud config is in nodepool's homedir so you need -H | 17:22 |
| clarkb | ya | 17:22 |
| fungi | sudo -H -u nodepool nodepool list | 17:22 |
| fungi | wfm on nl01 | 17:22 |
| frickler | oh, right, I did that as user nodepool. sorry for the confusion | 17:23 |
| *** kopecmartin is now known as kopecmartin|off | 17:23 | |
| * frickler should go to bed instead of working late :-/ | 17:24 | |
| *** dave-mccowan has quit IRC | 17:27 | |
| *** ccamacho has quit IRC | 17:29 | |
| *** lpetrut has quit IRC | 17:30 | |
| fungi | frickler: i concur, sleep is good. before you go though, any opinion on resizing the paste.o.o trove instance based on my findings above? | 17:32 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Document the OpenDev docker image jobs https://review.openstack.org/643108 | 17:32 |
| *** Swami has joined #openstack-infra | 17:33 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:33 | |
| *** jpich has quit IRC | 17:35 | |
| *** bhavikdbavishi has quit IRC | 17:37 | |
| frickler | fungi: I don't have an overview of our quota, but given the usage numbers you listed, that seems like a reasonable step | 17:39 |
| corvus | ++ | 17:39 |
| fungi | yeah, the quota's not so tight we can't absorb another 2gb ram and 5gb disk | 17:39 |
| fungi | thanks! | 17:39 |
| fungi | i'll snapshot and resize now | 17:39 |
| *** gfidente has quit IRC | 17:39 | |
| clarkb | https://opendev.org/openstack/ansible-role-tripleo-sahara is a grouped project that now points to tripleo on launchpad. However my normal browser seems to have cached the storyboard content. Going to incognito mode fixes it | 17:39 |
| clarkb | a hard refresh does not fix it | 17:40 |
| clarkb | so unsure what is being cached there | 17:40 |
| *** gfidente has joined #openstack-infra | 17:40 | |
| fungi | more service worker madness, perhaps? | 17:40 |
| clarkb | maybe? | 17:40 |
| clarkb | bug given that I'm pretty reasonably satisfied my changes are working and now I just have to wait for the playbook to finish | 17:40 |
| corvus | clarkb: i'm unlikely to have had it cached; it takes me to lp in firefox | 17:40 |
| clarkb | corvus: perfect | 17:40 |
| clarkb | so ya now we wait and that particular task should be able to be marked off the list | 17:41 |
| clarkb | and maybe add a "figure out caches" task? | 17:41 |
| *** eharney has joined #openstack-infra | 17:41 | |
| corvus | clarkb, fungi, mordred, EmilienM, dmsimard, tristanC: documentation for the opendev container image jobs: http://logs.openstack.org/08/643108/1/check/tox-docs/74dc39b/html/docker-image.html | 17:42 |
| clarkb | corvus: ooh I'll read through that shortly | 17:42 |
| *** jpena is now known as jpena|off | 17:42 | |
| anteaya | I get what looks like a gitea page from https://opendev.org/openstack/ansible-role-tripleo-sahara | 17:42 |
| anteaya | firefox | 17:43 |
| fungi | anteaya: and then click the issues link | 17:43 |
| fungi | and where does it send you? | 17:43 |
| fungi | (storyboard vs launchpad) | 17:43 |
| anteaya | launchpad | 17:43 |
| anteaya | sorry I missed the relevant bit | 17:43 |
| fungi | cool, that's what we want for projects which aren't yet using storyboard | 17:45 |
| corvus | dmsimard, tristanC, tobiash: next i'm planning to write docs for zuul-jobs describing how to create jobs like the opendev ones described at http://logs.openstack.org/08/643108/1/check/tox-docs/74dc39b/html/docker-image.html but the short version is "run a registry in whatever way you see fit and copy the job defs in opendev/base-jobs and add credentials". with that and the docs i just pushed up, | 17:45 |
| corvus | you should get a pretty good idea of the system. | 17:45 |
| corvus | SpamapS: ^ | 17:45 |
| anteaya | okay good to know, thank you | 17:46 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Document the OpenDev docker image jobs https://review.openstack.org/643108 | 17:47 |
| *** ginopc has quit IRC | 17:47 | |
| clarkb | corvus: fungi: mordred we probably want to think about the launchpad to storyboard transition in the context of the that playbook too. I'm guessing we don't want to run the full thing for each transition (it isn't fast) so maybe we want another fixup playbook that can take a project name as a var and only update that project? | 17:48 |
| *** cristoph has joined #openstack-infra | 17:48 | |
| tobiash | corvus: awesome :) | 17:48 |
| *** jamesmcarthur has quit IRC | 17:49 | |
| fungi | clarkb: yeah, it could just become part of the migration process since that's already manually performed anyway | 17:50 |
| *** wolverineav has quit IRC | 17:50 | |
| openstackgerrit | Merged openstack-dev/pbr master: Change openstack-dev to openstack-discuss https://review.openstack.org/625390 | 17:51 |
| clarkb | infra-root last warning. I'd like to commence https://etherpad.openstack.org/p/afs-dbserver-trusty-to-xenial on afsdb02 after I eat some lunch today | 17:51 |
| clarkb | please erad through the plan and offer feedback or raise the alarm if you think I've missed something important | 17:52 |
| anteaya | I like erad as a verb | 17:52 |
| *** ianychoi has quit IRC | 17:55 | |
| *** wolverineav has joined #openstack-infra | 17:55 | |
| *** panda|rover is now known as panda|rover|off | 17:57 | |
| mordred | clarkb: yes. I think a single-project fixup playbook is a great idea ... maybe we should refactor the current role into a couple of roles with the current one using include_role - so that we could make a new one that only calls the right sub-role or something | 17:59 |
| clarkb | mordred: ++ or something :) | 17:59 |
| fungi | paste.o.o trove instance snapshot finished, disk resize underway | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| *** mattw4 has quit IRC | 18:00 | |
| mordred | infra-root: I'm back from the doctor but may be slow on the draw for teh rest of the day | 18:01 |
| *** martinkennelly has quit IRC | 18:02 | |
| clarkb | corvus: my only suggestion is that http://logs.openstack.org/08/643108/1/check/tox-docs/74dc39b/html/docker-image.html#job-opendev-build-docker-image seems to replace http://logs.openstack.org/08/643108/1/check/tox-docs/74dc39b/html/docker-image.html#job-opendev-buildset-registry if you use it. Maybe make that more explicit? | 18:02 |
| fungi | mordred: hope they didn't take your trigger finger! | 18:02 |
| *** mattw4 has joined #openstack-infra | 18:03 | |
| fungi | paste.o.o trove instance disk resize finished, ram resize underway | 18:03 |
| mordred | fungi: I think it's a felony in texas to mess with any body part that could be used to operate a gun | 18:03 |
| Shrews | mordred: have you tried NOT going to the doctor? | 18:03 |
| fungi | (ram resize includes an instance restart, so there may be a bit of a blip for a the paste service) | 18:04 |
| mordred | Shrews: that would have been helpful advice BEFORE I went to the doctor | 18:04 |
| fungi | only if you're stuck in linear time | 18:05 |
| *** ricolin has quit IRC | 18:05 | |
| mordred | fungi: in soviet russia, linear time is stuck in you? | 18:08 |
| *** xek_ has joined #openstack-infra | 18:08 | |
| fungi | you in stuck is time linear, russia soviet in | 18:09 |
| mordred | malkovich malkovich | 18:09 |
| *** xek has quit IRC | 18:10 | |
| fungi | paste.o.o trove instance ram resize finished, please test the service out and make sure it's still working | 18:11 |
| fungi | ideally, performance will also be better | 18:12 |
| fungi | though it may take the server a moment to realize its database socket went away :/ | 18:12 |
| clarkb | ya new paste isn't working currently | 18:12 |
| clarkb | restarting the service might jump start that timeout | 18:12 |
| fungi | okay, it just returned content for me a moment ago now | 18:12 |
| clarkb | oh it responded | 18:12 |
| EmilienM | corvus: ack, awesome | 18:12 |
| clarkb | http://paste.openstack.org/show/747729/ works for me | 18:13 |
| fungi | yeah, pulling up random pastes from my history are returning content, and perhaps more quickly than before | 18:13 |
| fungi | at the moment the rackspace dashboard shows memory used has now jumped to 4gb, but it may take some time to settle out, i'm not sure | 18:14 |
| fungi | or it may be the case that mysql is configured to preallocate that amount of ram? | 18:15 |
| SpamapS | fungi: yes, it will allocate the innodb buffer pool entirely at startup | 18:15 |
| SpamapS | and that's usually configured for 50-80 percent of RAM. | 18:16 |
| clarkb | mordred: openstack/deb-oslotest seems to have just failed to set the default branch | 18:16 |
| SpamapS | (not sure what rax trove defaults to) | 18:16 |
| clarkb | it is an empty repo | 18:16 |
| fungi | SpamapS: thanks, that helps! | 18:17 |
| fungi | so the fact that they reported the trove instance using all of its ram may not have been an actual indicator of insufficient resource allocation, but it does seem to have helped performance | 18:18 |
| *** priteau has quit IRC | 18:18 | |
| mordred | clarkb: oh- right. I think that's one of teh things that's in projects.yaml that we shoudl remove | 18:18 |
| clarkb | mordred: looking at it in gerrit it is a read only project with a master branch | 18:19 |
| clarkb | so I am a bit confused about why it doesn't have content in gitea | 18:19 |
| mordred | clarkb: oh. ok. then I'm not sure what's up with it then | 18:19 |
| clarkb | oh I wonder if we need to trigger replication for it | 18:19 |
| clarkb | if it was hidden before then maybe we never replicated it on start up | 18:19 |
| clarkb | and since no changes go to it it isn't being replicated? | 18:19 |
| clarkb | any objection to me triggering replication for it? | 18:19 |
| *** gfidente has quit IRC | 18:19 | |
| mordred | go for it | 18:20 |
| mordred | fungi: malloc() is expensive, thus the innodb pre-allocation of the buffer pool | 18:21 |
| fungi | makes sense, sure | 18:22 |
| clarkb | mordred: that doesn't seem to hae changed anything | 18:22 |
| fungi | #status log snapshotted and increased the paste-mysql-5.6 trove instance from 5gb disk/2gb ram to 10gb disk/4gb ram | 18:22 |
| openstackstatus | fungi: finished logging | 18:22 |
| *** mattw4 has quit IRC | 18:25 | |
| clarkb | I can clone that repo from git.openstack.org so there is content there | 18:26 |
| clarkb | are read only projects maybe not replicated and the git.o.o content is from before we went read only? | 18:26 |
| fungi | well, any content would need to be from before they were read-only | 18:26 |
| clarkb | ya I'm curious why we aren't replicating the content to gitea now that I've explicitly told it replicate via ssh command | 18:27 |
| clarkb | any objection to me setting it to active and trying the explicit replicate command again? | 18:27 |
| corvus | clarkb: i tried to make that explicit in http://logs.openstack.org/08/643108/1/check/tox-docs/74dc39b/html/docker-image.html#a-repository-with-only-producers but maybe you're saying i should clarify that in the job documentation itself? | 18:27 |
| clarkb | corvus: yes exactly | 18:28 |
| *** pcaruana has quit IRC | 18:29 | |
| fungi | clarkb: maybe good to look at more examples. openstack/deb-oslotest sticks in my mind as a repo there was something wrong with in the beginning. how about openstack/instack? that was retired a month ago today | 18:29 |
| clarkb | fungi: instack is active in gerrit and its retired readme content is replicated to gitea | 18:30 |
| *** wolverineav has quit IRC | 18:30 | |
| fungi | oh, it never got an acl change i guess? | 18:30 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/instack https://opendev.org/openstack/instack | 18:31 |
| clarkb | ya https://review.openstack.org/#/admin/projects/openstack/instack,access looks like a non retired acl | 18:31 |
| *** wolverineav has joined #openstack-infra | 18:31 | |
| fungi | ahh, right, that one's only retired on master but stable branches are open for continued development | 18:31 |
| fungi | bad example | 18:31 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Document the OpenDev docker image jobs https://review.openstack.org/643108 | 18:31 |
| *** wolverineav has quit IRC | 18:31 | |
| *** wolverineav has joined #openstack-infra | 18:31 | |
| clarkb | fungi: maybe another deb-foo? | 18:31 |
| fungi | clarkb: how about openstack/gantt? that was retired in january | 18:32 |
| clarkb | looking (fwiw deb-nova is in the instack state looks like) | 18:32 |
| fungi | i'm dubious of the deb-.* repos as a whole | 18:32 |
| clarkb | gantt is in the instackt state too | 18:33 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/gantt https://review.openstack.org/#/admin/projects/openstack/gantt,access https://opendev.org/openstack/gantt | 18:33 |
| *** Emine has joined #openstack-infra | 18:33 | |
| fungi | huh, did something break with our application of acls? | 18:34 |
| fungi | https://review.openstack.org/630162 switched the acl-config for openstack/gantt to /home/gerrit2/acls/openstack/retired.config | 18:34 |
| corvus | i'm having trouble following this conversation.... clarkb, can you recap? | 18:34 |
| clarkb | corvus: deb-oslotest failed to have its default branch set by our gitea fixup playbook this is because there is no repo content | 18:35 |
| clarkb | corvus: looking at Gerrit config it is read only and retired. Running an explicit replication start openstack/deb-oslotest did not replicate the data | 18:35 |
| *** ykarel|away has quit IRC | 18:35 | |
| clarkb | corvus: fungi's suggestion was to look at other projects that are retired and I think we may have stumbled across an issue with us not applying the retired.confg acl config in gerrit | 18:35 |
| fungi | separately, in trying to find other examples of read-only projects to compare against, we're finding them all set to active so far | 18:36 |
| fungi | yeah, that | 18:36 |
| corvus | interesting, we encounted one similar situation because a project was set to *hidden* but that's not our normal process | 18:36 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/cloudv-ostf-adapter https://review.openstack.org/#/admin/projects/openstack/cloudv-ostf-adapter,access https://opendev.org/openstack/cloudv-ostf-adapter is in the deb-oslotest gerrit config situation but with content in gitea | 18:36 |
| clarkb | so I don't think the read only state of the project is at fault here | 18:36 |
| clarkb | so there are two issues we are looking at 1) fixing deb-oslotest and getting fixup playbook to run to completion 2) setting retired.confg acls on projects | 18:37 |
| clarkb | deb-oslotest does not show up in the gerrit error log either | 18:37 |
| fungi | on review.o.o, /home/gerrit2/review_site/git/openstack/deb-oslotest.git/HEAD contains "ref: refs/heads/master" so it's not a default branch problem i don't think | 18:37 |
| corvus | k. i'll help dig into deb-oslotest. i'll log into a gitea host and look for replication errors on its side, as well as from gerrit | 18:37 |
| corvus | clarkb: nevermind you got the gerrit side already :) | 18:38 |
| clarkb | corvus: thanks | 18:38 |
| clarkb | corvus: I ran clarkb@review01:~$ grep deb-oslotest ../gerrit2/review_site/logs/error_log and got nothing back | 18:38 |
| corvus | clarkb: oh, then we should still check the replication log on gerrit | 18:38 |
| clarkb | ah there is more than one log got it | 18:38 |
| corvus | it's separate; i'm unsure if problems there show up in error_log | 18:38 |
| clarkb | Failed replicate of refs/tags/1.5.3 to ssh://git@gitea02.opendev.org:222/openstack/deb-oslotest.git, reason: pre-receive hook declined | 18:39 |
| clarkb | there are lines like that, so that is progress maybe | 18:39 |
| fungi | i don't see any hooks in that repo on the gerrit fs at least | 18:40 |
| clarkb | as a potential third issue the fixup playbook has not exited after those errors and isn't reporting any new content to my console | 18:40 |
| clarkb | fungi: I think they are the hooks on the gitea side? | 18:40 |
| *** priteau has joined #openstack-infra | 18:40 | |
| corvus | wow, we should install "less" on our servers | 18:41 |
| fungi | yeah, just wondering if gerrit also tries to replicate hooks | 18:41 |
| fungi | and whether it was trying to or had pushed one during replication if so | 18:41 |
| fungi | but i don't see one it would have tried to put in gitea, so if there's a pre-receive hook in gitea that's not how it got there | 18:41 |
| clarkb | I want to say C git at least hooks are always opt in by the client | 18:41 |
| fungi | possible gitea has some default hooks and something about this repo is running afoul of one? | 18:42 |
| corvus | fungi: it may be that gitea ingestion includes implementation of something that looks like a hook to git | 18:43 |
| corvus | gitea internally handles receiving commits (just like gerrit) | 18:43 |
| fungi | sure, also seems like a possibility | 18:43 |
| *** priteau has quit IRC | 18:44 | |
| fungi | i.e. it's "default hooks" aren't in files on the filesystem, just some tasks it performs internally | 18:44 |
| corvus | 2019/03/13 18:20:55 [...io/gitea/cmd/hook.go:109 runHookPreReceive()] [F] retrieve protected branches information failed: parse https://localhost:3000/api/internal/branch/620/refs/tags/debian/1%!.(MISSING)0-2: invalid URL escape "%!"(MISSING) | 18:44 |
| *** wolverineav has quit IRC | 18:44 | |
| corvus | that's from /var/gitea/logs/hooks/pre-receive.log on gitea01 | 18:45 |
| fungi | ooh! | 18:45 |
| fungi | did we import a funky tag? | 18:45 |
| corvus | i don't know if that's our repo in question, but it's suggestive :) | 18:45 |
| corvus | there is a "debian/1%1.6.0-2" according to gerrit | 18:45 |
| fungi | i wonder where it comes from, as /home/gerrit2/review_site/git/openstack/deb-oslotest.git/refs/tags/ is empty on review.o.o | 18:46 |
| corvus | fungi: could be in packed-refs | 18:46 |
| clarkb | I wonder if that got into the repo with an old git version then newer git decided you have to do %% or whatever the escape is | 18:46 |
| fungi | aha, i keep forgetting about packed refs | 18:46 |
| clarkb | idea: lets create a new tag then delete that one? | 18:46 |
| fungi | clarkb: yeah, that rings a bell. there were some advisories about malformed references in recent history | 18:46 |
| *** kjackal has quit IRC | 18:47 | |
| *** jamesmcarthur has joined #openstack-infra | 18:47 | |
| mordred | gitea is talking about url escape though - it's possible that gitea is apply a url escape routine to the ref name - and nothing about it is invalid git and it's just a gitea bug doing ingestion processing | 18:47 |
| clarkb | debian/1.6.0-2 would line up with debian/1.6.0-1 which already exists then delete the % ref? | 18:47 |
| *** jtomasek has quit IRC | 18:48 | |
| corvus | mordred: i'm inclined to agree. if so, whatever else we have going on here, we may have a gitea bug. | 18:48 |
| mordred | yeah | 18:48 |
| *** gmann is now known as gmann_afk | 18:48 | |
| mordred | like - let's delete the tag - but it also might be worth filing a bug - and that's a pretty easy reproduction case | 18:48 |
| clarkb | `git check-ref-format` is apparently the source of truth for git | 18:48 |
| corvus | given the state of that repo though, i also think it would be okay to "fix" this by replacing that tag. | 18:49 |
| mordred | ++ | 18:49 |
| clarkb | check-ref-format doesnt' error on that tag name for me | 18:49 |
| clarkb | so ya likely a gitea bug | 18:49 |
| *** mattw4 has joined #openstack-infra | 18:51 | |
| corvus | if i try to create that tag in try.gitea.io web ui, i get " TagName must be a well-formed Git reference name." | 18:52 |
| corvus | it seems that gitea web ui has some validation that goes beyond git, and i bet if i did a git push with the tag, i'd hit the same error we see in replication | 18:53 |
| corvus | still, i think gitea should be able to handle that since git does | 18:53 |
| corvus | i'll file a bug, unless anyone else has started on that? | 18:53 |
| clarkb | `git checkout 9bf5d27b53bde8b5a2856ca44e395992379caad2 && git tag -s debian/1.6.0-2 && git push gerrit debian/1.6.0-2 && git push gerrit :debian/1%1.6.0-2` look right? | 18:54 |
| clarkb | corvus: I've not started filing a bug (was working on sequence of things to replace the tag | 18:54 |
| *** mattw4 has quit IRC | 18:56 | |
| *** mattw4 has joined #openstack-infra | 18:57 | |
| *** kjackal has joined #openstack-infra | 18:58 | |
| mordred | clarkb: yes - that looks right to me | 18:58 |
| mordred | clarkb: although I think you could also do "git checkout debian/1%1.6.0-2" as step 1 | 18:58 |
| clarkb | mordred: ah yup. I'm ready to do the above commands shoudl I go ahead and do them? | 18:59 |
| clarkb | wow ok so these repos | 19:00 |
| clarkb | turns out if you git review -s the gerrit remote you get is for openstack/oslotest not openstack/deb-oslotest | 19:00 |
| clarkb | (I haven't psuehd anything yet, and happy I checked that. Amazing) | 19:01 |
| *** e0ne has joined #openstack-infra | 19:01 | |
| clarkb | replace the git pushes above with `git push ssh://cboylan@review.openstack.org:29418/openstack/deb-oslotest.git debian/1.6.0-2 && git push ssh://cboylan@review.openstack.org:29418/openstack/deb-oslotest.git :debian/1%1.6.0-2` | 19:02 |
| *** mattw4 has quit IRC | 19:02 | |
| mordred | ++ | 19:03 |
| corvus | clarkb, mordred, fungi: https://github.com/go-gitea/gitea/issues/6321 | 19:03 |
| *** mattw4 has joined #openstack-infra | 19:03 | |
| clarkb | corvus: thanks | 19:03 |
| clarkb | I'm pushing the tag now | 19:03 |
| fungi | clarkb: i'm okay with the proposed solution since it's a dead repo anyway | 19:04 |
| *** zzzeek has quit IRC | 19:04 | |
| fungi | but agree it seems like a bug in gitea | 19:04 |
| clarkb | ! [remote rejected] debian/1.6.0-2 -> debian/1.6.0-2 (prohibited by Gerrit) | 19:04 |
| clarkb | I'm guessing I need to make the repo active? | 19:04 |
| fungi | yes, it'll have to be active and have a viable acl, i think | 19:05 |
| clarkb | I can make it active the nset it back to read only if we still like that plan | 19:05 |
| clarkb | fungi: I'm bypassing the acl via project bootstrappers fwiw | 19:05 |
| clarkb | ok pushes are done. setting repo back to read only | 19:06 |
| clarkb | and will trigger replication again | 19:07 |
| fungi | yeah, as long as the acl properly inherits from all-projects that works | 19:07 |
| *** zzzeek has joined #openstack-infra | 19:07 | |
| clarkb | https://opendev.org/openstack/deb-oslotest | 19:08 |
| clarkb | I'm going to grab lunch now that that is happy. We still need to figure out retirement acl updates and my gitea fixup playbook still hasn't exited which seems like something we don't want either | 19:10 |
| clarkb | mordred: ^ any ideas on that? its running as root in screen on bridge if you want to take al ook | 19:10 |
| clarkb | #status log Replaced deb-oslotest's debian/1%1.6.0-2 tag as the % is making gitea unhappy and the project is retired. Gitea bug: https://github.com/go-gitea/gitea/issues/6321 New tag: debian/1.6.0-2 | 19:12 |
| openstackstatus | clarkb: finished logging | 19:12 |
| *** emine__ has joined #openstack-infra | 19:12 | |
| *** Emine has quit IRC | 19:13 | |
| *** wolverineav has joined #openstack-infra | 19:15 | |
| *** wolverineav has quit IRC | 19:22 | |
| *** luizbag_ has quit IRC | 19:30 | |
| *** luizbag has joined #openstack-infra | 19:30 | |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Pagination error in active/merged/invalid stories in projects resolved. https://review.openstack.org/642224 | 19:38 |
| clarkb | infra-root I'm going to stop the current running playbook via ^C, start it again, the start looking at afsdb02 | 19:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Disable wheel mirror updates https://review.openstack.org/643146 | 19:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "Disable wheel mirror updates" https://review.openstack.org/643147 | 19:42 |
| mordred | clarkb: ++ | 19:42 |
| clarkb | infra-root can I get reviews on https://review.openstack.org/643146 to disable wheel mirror updates. I am disabling the other updates now | 19:43 |
| *** wolverineav has joined #openstack-infra | 19:45 | |
| corvus | clarkb, mordred, fungi, AJaeger: regarding the opendev base-jobs documentation.... we need to find a place to publish it :) | 19:52 |
| clarkb | docs.opendev.org then host that off afs maybe? | 19:52 |
| clarkb | we've already got the afs volume (though it needs the RO spare) | 19:53 |
| corvus | which probably means answering the question "should opendev have a general purpose publishing site for any project?" | 19:53 |
| corvus | or do we just set something up for publishing dockes for opendev itself? | 19:53 |
| corvus | that was a weird way to spell docs | 19:53 |
| clarkb | https://github.com/go-gitea/gitea/issues/302 suggests "caddy" as a tool to enable it if we want something like github pages | 19:55 |
| clarkb | I think there is value in supporting that. But we need to force projects onto their own domains | 19:56 |
| clarkb | (I wonder if that would be a reason to host at git.opendev.org instead of opendev.org | 19:56 |
| corvus | clarkb: or we register opendevusercontent.org :) | 19:58 |
| *** mattw4 has quit IRC | 19:58 | |
| mordred | or do wildcards and allow for nova.openstack.opendev.org ? | 19:59 |
| corvus | do i recall correctly that the concern with cookies and domains is that arbitrary content at foo.opendev.org receives the cookies for opendev.org ? | 20:00 |
| mordred | ah. maybe so | 20:00 |
| *** mattw4 has joined #openstack-infra | 20:00 | |
| mordred | well - I don't thnik we should do a gh-pages like service | 20:00 |
| mordred | we have zuul | 20:00 |
| mordred | we don't need a service that knows how to publish static content | 20:00 |
| clarkb | corvus: yes I think that is the concern | 20:00 |
| clarkb | whereas git.opendev.org and foo.opendev.org woudl not have that hierarchical relationship aiui | 20:01 |
| mordred | but I don't have a good suggestion of where to put project-tied repo content | 20:01 |
| clarkb | as soon as my wheel job disable chagne merges my next step is going to be to disable the afs processes on afsdb02.o.o. Then I'll wait a few minutes where we can make sure clients still work before proceeding | 20:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable wheel mirror updates https://review.openstack.org/643146 | 20:02 |
| dulek | clarkb: Do we have infra mirrors of K8s releases? In particular I mean binary releases that are downloaded by kubetest --extract, so "staged build from gcs", as it's documentation states. | 20:02 |
| clarkb | dulek: no | 20:02 |
| clarkb | I don't think we mirror any of the docker or k8s binaries or packages | 20:03 |
| clarkb | unless they come from a distro | 20:03 |
| dulek | clarkb: Okay, will get back to you if downloading that from GCS will be problematic on gate VM's. | 20:04 |
| dulek | Hopefully it wont. :) | 20:04 |
| mordred | looks like if it does turn out we need to mirror some location for that- kubetest --extract supports the env var "KUBERNETES_RELEASE_URL" | 20:05 |
| clarkb | infra-root afsdb02 has afs services disabled now. I'm going to let it sit like that for a few minutes and then check that docs.openstack.org and zuul-ci.org and mirror.rax.dfw.openstack.org still serve content | 20:05 |
| mordred | so it should be possible to tell it to use a mirror if we grew one | 20:05 |
| clarkb | corvus: ^ maybe you want to do tests from your local machine (I currently don't have working afs kernel modules. it gets mad about some piocontrol or omsething) | 20:06 |
| clarkb | though the mirrors and files hosts should be pretty good canaries | 20:06 |
| corvus | clarkb: i'm still able to browse /afs/openstack.org | 20:08 |
| clarkb | the sites I listed are all working still as well | 20:09 |
| clarkb | I am going to proceed with my plan as documented | 20:09 |
| *** dklyle has quit IRC | 20:09 | |
| corvus | mordred, clarkb: i don't want to distract from the afs maint anymore, but reading the cookie spec, i think we might be okay. i think we should look into it more after the maint. | 20:11 |
| *** luizbag has quit IRC | 20:12 | |
| *** mattw4 has quit IRC | 20:13 | |
| *** wolverineav has quit IRC | 20:15 | |
| *** dklyle has joined #openstack-infra | 20:16 | |
| *** wolverineav has joined #openstack-infra | 20:16 | |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Stars should be next to the name of the entity https://review.openstack.org/643153 | 20:17 |
| clarkb | upgrade is to the point of building dkms modules | 20:23 |
| clarkb | which is a long waiting period before answering a few more questions then rebooting | 20:23 |
| clarkb | When it comes back I | 20:24 |
| clarkb | er | 20:24 |
| clarkb | after that first reboot I'll correct apt things and make puppet happen then disable afs again and do another set of reboots to amke sure we don't have to hard reboot it like the others | 20:24 |
| *** mattw4 has joined #openstack-infra | 20:25 | |
| *** lpetrut has joined #openstack-infra | 20:27 | |
| *** jamesmcarthur has quit IRC | 20:29 | |
| *** lpetrut has quit IRC | 20:31 | |
| clarkb | post do-release-upgrade reboot is happening now | 20:37 |
| clarkb | bos status says services on afsdb02 are running normally post reboot. I'm going to clean up apt and puppet now. Then line up some more reboots | 20:38 |
| *** e0ne has quit IRC | 20:38 | |
| clarkb | ok first reboot off of new kernel happening in just a moment. | 20:40 |
| *** gmann_afk is now known as gmann | 20:41 | |
| *** e0ne has joined #openstack-infra | 20:41 | |
| clarkb | and it did not hang like the fileservers | 20:42 |
| clarkb | I am going to reenable puppet on afsdb02 now | 20:42 |
| *** mattw4 has quit IRC | 20:43 | |
| *** mattw4 has joined #openstack-infra | 20:43 | |
| clarkb | infra-root afsdb02 ^ is now running xenial and the upgrade is complete pending a successful puppet run | 20:43 |
| clarkb | The upgrade process went exactly as expected in my document as far as question prompts | 20:43 |
| clarkb | service seems to be running happily | 20:44 |
| clarkb | Given the lack of surprises I'll plan to upgrade afsdb01 as soon as afsdb02 has a good puppet run | 20:44 |
| clarkb | corvus: mordred now might be a good time to talk cookies? | 20:44 |
| clarkb | I like chocolate chip | 20:44 |
| *** e0ne has quit IRC | 20:45 | |
| mordred | mmm. chocolate chip are delicious | 20:45 |
| mordred | they go nicely into cookie dough inside of ice cream | 20:45 |
| clarkb | ++ | 20:46 |
| *** weshay is now known as other_guys | 20:46 | |
| *** other_guys is now known as other_guy | 20:46 | |
| mordred | clarkb: yeah - I agree with what you said above - it seems that cookies are scoped to a host, not a domain | 20:47 |
| mordred | although it does seem like we COULD set cookies with a domain scope by prefixing with a . | 20:48 |
| mordred | clarkb: oh - wait | 20:48 |
| clarkb | mordred: stack overflow says none of the browsers care about the leading . now and all follow rfc6265 | 20:48 |
| clarkb | and that says if you set a domain then it is sent for all subdomains too | 20:48 |
| mordred | clarkb: yeah | 20:49 |
| clarkb | so depends on what our gitea cookies look like | 20:49 |
| mordred | but I think still if you don't add a domain= to the cookie | 20:49 |
| mordred | it should still be host scoped | 20:49 |
| corvus | back | 20:50 |
| corvus | https://tools.ietf.org/html/rfc6265#section-4.1.2.3 | 20:50 |
| corvus | https://tools.ietf.org/html/rfc6265#section-5.3 step 6 | 20:50 |
| corvus | https://tools.ietf.org/html/rfc6265#section-8.6 | 20:50 |
| clarkb | the cookies I see in firefox have domain=opendev.org | 20:50 |
| clarkb | oh then they set hostonly=true | 20:50 |
| corvus | those are the relevant rfc links i found | 20:50 |
| corvus | clarkb: and yes -- gitea does not set the domain attribute, which means (according to the first and second links) that it should be host-only, and therefore only supplied back to opendev.org, not foo.opendev.org | 20:51 |
| mordred | cool. so we could do a docs.opendev.org if we wanted to | 20:52 |
| corvus | https://security.stackexchange.com/questions/91427/security-implications-of-providing-users-subdomains-on-my-domain elaborates on what's described in the third link there | 20:52 |
| mordred | and let people publish in a zuul job to a project-limited location (like we do now for tarballs) pretty easily | 20:52 |
| corvus | but i believe that macaron employs the csrf prevention technique described there (in that it resets the csrf token on login and logout) so that's probably okay too | 20:52 |
| corvus | mordred: i'm inclined to agree | 20:53 |
| *** yamamoto has quit IRC | 20:53 | |
| corvus | (oh, sorry, macaron is the web framework used by gitea) | 20:53 |
| clarkb | the way firefox renders the cookie data for me is weird it totally shows domain=opendev.org implying the domain is set | 20:53 |
| clarkb | but then there is another field called hostonly set to true | 20:53 |
| clarkb | end result is ya Ithink this is safe | 20:54 |
| corvus | clarkb: i think you're looking at cookie storage, not transport? | 20:54 |
| clarkb | corvus: yes | 20:54 |
| corvus | clarkb: the steps described in rfc6265 relate to how to translate cookie transport to storage, and they do indeed say that if you supply no domain attribute in transport, you should set the domain to the host and set host-only to true. | 20:54 |
| clarkb | got it | 20:55 |
| corvus | clarkb: so your observation is consistent with what i observe in transport using the dev tools -- the set-cookie header does not have "domain=" | 20:55 |
| *** other_guy is now known as weshay | 20:55 | |
| clarkb | is there concern about going the other direction. Eg something in docs.opendev.org setting a cookie with domain=opendev.org (I don't know if it is allowed to) then having that get loaded by gitea? | 20:55 |
| corvus | clarkb: sorry i meant to say steps described in rfc6265 section 5.3 (the second link above) | 20:55 |
| corvus | clarkb: yes, that's described in 8.6 and, i think, aiui, mitigated by the csrf | 20:56 |
| corvus | the csrf token that is :) | 20:56 |
| mordred | ++ | 20:56 |
| corvus | if this is passing the sniff test with folks here in channel, maybe we should write it up real quick and ask a smart keystone person to look it over? :) | 20:57 |
| clarkb | ++ | 20:58 |
| clarkb | we haev three cookies from gitea, the csrf token, i_like_gitea, and lang. I guess csrf verifies the two others? | 20:59 |
| corvus | here's the docs, though i haven't digested them yet: https://go-macaron.com/docs/middlewares/csrf | 21:00 |
| clarkb | I guess the subdomain setting cookie for domain=opendev.org would be an issue if we used git.opendev.org or not | 21:03 |
| clarkb | so the question is more do we need a completely different domain than if git.opendev.org would work | 21:03 |
| corvus | clarkb: that's my understand too | 21:04 |
| *** snevi has joined #openstack-infra | 21:04 | |
| clarkb | cool seems like we have all largely come to the same conclusions then. NOw just a matter of understanding if the csrf cookie stands up to this appropriately | 21:06 |
| *** IvensZambrano has quit IRC | 21:06 | |
| gmann | fungi: clarkb I am ready to go on these - https://review.openstack.org/#/c/639096/ https://review.openstack.org/#/c/641886/ | 21:08 |
| *** trown is now known as trown|outtypewww | 21:10 | |
| clarkb | gmann: +2 on https://review.openstack.org/#/c/639096/8 but left a note of somethign to watch | 21:10 |
| *** whoami-rajat has quit IRC | 21:11 | |
| clarkb | gmann: at the bottom of the diff at https://review.openstack.org/#/c/641886/2/zuul.d/zuul-legacy-jobs.yaml why did we add the xenial nodeset to legacy-tempest-dsvm-neutron-full-ocata? | 21:11 |
| clarkb | infra-root puppet just ran on afsdb02 successfully with no changes | 21:11 |
| clarkb | infra-root (and corvus in particular) should I go ahead and start on afsdb01? I can stop services there then wait again | 21:12 |
| gmann | clarkb: legacy-tempest-dsvm-neutron-full-ocata is stable job run on tempest master gate so need to use xenial | 21:12 |
| *** kgiusti has left #openstack-infra | 21:12 | |
| clarkb | gmann: got it | 21:12 |
| gmann | clarkb: RE on 639096, does not release pipeline take the master version ? | 21:13 |
| corvus | clarkb: yep, i'm standing by | 21:13 |
| clarkb | gmann: no beacuse tags aren't branch specific | 21:14 |
| clarkb | gmann: one thought there is to not worry about the legacy nodests on tag based pipelines | 21:14 |
| clarkb | ok stopping services on afsdb01 now | 21:14 |
| clarkb | will wait ~5 minutes then check afs hosted sites for cotent | 21:16 |
| *** mattw4 has quit IRC | 21:17 | |
| clarkb | gmann: there is also work in zuul to make it infer a branch | 21:18 |
| clarkb | but that isn't merged yet | 21:18 |
| clarkb | at least I don't think it is | 21:18 |
| corvus | Mar 13 14:18:13 fuligin kernel: [241552.785217] afs: Lost contact with volume location server 104.130.136.20 in cell openstack.org (code -1) | 21:18 |
| corvus | clarkb: i got that, and then my kernel recovered and tried afsdb02 and succeeded | 21:18 |
| *** mattw4 has joined #openstack-infra | 21:18 | |
| clarkb | corvus: awesome | 21:18 |
| corvus | so lgtm so far | 21:18 |
| *** wolverineav has quit IRC | 21:18 | |
| *** janki has quit IRC | 21:19 | |
| corvus | it took like... i dunno, 30 seconds? a while. | 21:19 |
| * fungi is catching up on xss/csrf/cors/cookie discussion and afs maintenance while working on dinner | 21:19 | |
| clarkb | fungi: gmann is ready to merge things too | 21:19 |
| fungi | gmann: i'll take one last look but we're probably good to merge those | 21:19 |
| mordred | fungi: I recommend dinner | 21:19 |
| fungi | and then we can merge mine once we see they're in | 21:19 |
| clarkb | corvus: docs is still there | 21:20 |
| corvus | i love how well-supported afs is in tcpdump: | 21:20 |
| clarkb | and our mirrors are still serving content | 21:20 |
| corvus | 14:18:17.562450 IP fuligin.afs3-callback > afsdb02.openstack.org.afs3-vlserver: rx data vldb call get-entry-by-name-u "mirror.debian-security" (60) | 21:20 |
| clarkb | based on that I think things are happy and I'm going to proceed | 21:20 |
| corvus | clarkb: ++ | 21:21 |
| mordred | corvus: this is a dumb question - but I thought afs didn't use tcp | 21:21 |
| clarkb | tcpdump captures udp too (despite the name) | 21:21 |
| gmann | clarkb: i see. these are the places that is being used - http://codesearch.openstack.org/?q=legacy-publish-openstack-artifacts&i=nope&files=&repos= | 21:21 |
| mordred | clarkb: well there we go then | 21:21 |
| *** wolverineav has joined #openstack-infra | 21:22 | |
| *** wolverineav has quit IRC | 21:24 | |
| *** wolverineav has joined #openstack-infra | 21:24 | |
| fungi | tcpdump even captures things which are not ip protocols at all | 21:24 |
| fungi | on some platforms, it's even used to capture things which aren't actually network traffic | 21:24 |
| fungi | openbsd's pf packet filter performs its logging to a virtual pflog0 "network" interface which you can stream in real time using tcpdump to sniff on it | 21:25 |
| mordred | fungi: bsd is weird | 21:25 |
| fungi | "keep bsd weird" | 21:25 |
| *** yamamoto has joined #openstack-infra | 21:26 | |
| clarkb | while I'm waiting for lots of text to scroll by: bionic has openafs 1.8.x | 21:28 |
| clarkb | so the next round of upgrades will be a bit more involved. I think we can have the fileservers be 1.8 with 1.6 db servers, but aiui we don't want to mix 1.6 and 1.8 db servers | 21:29 |
| clarkb | so that might be a case of take an outage, shutdown all db servers, upgrade one db server to 1.8 then upgrade the other | 21:29 |
| clarkb | we might be able to reduce the outage window size if we uninstall afs from one of the db servers, upgrade it to bionic then reinstall afs and do a juggle where we aren't waiting on the full upgrade to happen | 21:30 |
| clarkb | so a few minutes rather than ~40 minutes | 21:30 |
| clarkb | which isn't that bad | 21:30 |
| clarkb | (and now I'm going to stop thinking about problems 2 years from now) | 21:31 |
| corvus | i kind of want to write a zuul job that sets up a cell on xenial, then uses an ansible playbook to upgrade it to bionic. | 21:31 |
| fungi | gmann: i've approved both 639096 and 641886 just now. once we see those are applying we can approve 641897 which i've just removed my wip vote from | 21:33 |
| clarkb | I am in the build dkms modules part of the upgrade | 21:33 |
| gmann | fungi: thanks. | 21:34 |
| *** yamamoto has quit IRC | 21:35 | |
| mordred | corvus: if you wrote such a job, you be in a good position to offer some zuul testing to our openafs friends | 21:38 |
| corvus | mordred: oh indeed... | 21:39 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Migrating legacy base jobs nodeset from Xenial to Bionic https://review.openstack.org/639096 | 21:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move zuul-legacy-jobs.yaml jobs to Bionic https://review.openstack.org/641886 | 21:43 |
| fungi | gmann: ^ those are applied immediately to all new builds now | 21:43 |
| fungi | if thinks are looking good i'll get a second config-core reviewer to take a look at 641897 and hopefully approve it next | 21:44 |
| * mnaser is around | 21:44 | |
| clarkb | I can look as soon as I'm done with this upgrade. Getting close to the end then I clean up apt, install puppet, and do secnd reboot for sanity | 21:44 |
| fungi | clarkb: you already +2'd it anyway | 21:45 |
| gmann | fungi: +1, we are good to go seems. sent the last email about those are merged | 21:45 |
| fungi | mnaser: sounds like 641897 is safe to approve in that case | 21:47 |
| * mnaser holds breath | 21:48 | |
| mnaser | done, be good to watch the gates | 21:48 |
| clarkb | second reboot was fine on afsdb01, afsdb02 reports the status of afsdb01 is running normally. I am reenabling puppet now | 21:49 |
| clarkb | and we can be done with these upgrades for now | 21:49 |
| clarkb | #status log Upgraded afsdb01 and afsdb02 servers to Xenial from Trusty. | 21:50 |
| openstackstatus | clarkb: finished logging | 21:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Document how to build a buildset registry https://review.openstack.org/643167 | 21:50 |
| clarkb | infra-root I am going to reenable afs publishing things again on mirror-update and afsdb01. And will have approved https://review.openstack.org/#/c/643147/1 | 21:52 |
| openstackgerrit | Merged opendev/base-jobs master: Switch default nodeset to ubuntu-bionic https://review.openstack.org/641897 | 21:53 |
| clarkb | mirror-update chagnes should begin in about 10 minutes | 21:54 |
| clarkb | and the docs vos releases are every 5 minutes | 21:54 |
| clarkb | fwiw my second run of the correct gitea configs playbook is still running | 21:55 |
| clarkb | hasn't failed yet though | 21:55 |
| *** tosky has joined #openstack-infra | 21:56 | |
| fungi | mnaser: thanks! will keep an eye on it while eating dinner now | 21:56 |
| *** rcernin has joined #openstack-infra | 21:57 | |
| *** mattw4 has quit IRC | 22:01 | |
| *** mattw4 has joined #openstack-infra | 22:01 | |
| *** rh-jelabarre has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable wheel mirror updates" https://review.openstack.org/643147 | 22:02 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Remove trusty testing flag from nodes that are xenial https://review.openstack.org/643168 | 22:03 |
| clarkb | infra-root ^ cleanup of some of our uneeded trusty testing | 22:03 |
| clarkb | I think that means I should haev plenty of time tomorrow to do puppet4 upgrade things \o/ | 22:04 |
| mordred | clarkb: woot | 22:05 |
| clarkb | deb-python-mistralclient has caused gitea fixup to fail | 22:06 |
| clarkb | I'm looking into it now | 22:06 |
| clarkb | I wonder if we can easily scan all the repos for no content in gitea? | 22:06 |
| clarkb | curl | grep no content or something | 22:06 |
| *** kjackal has quit IRC | 22:07 | |
| clarkb | for project in projects.yaml ; do curl opendev.org/$project | grep 'This repository does not contain any content.' && echo $project ; done | 22:07 |
| clarkb | maybe? | 22:07 |
| clarkb | deb-python-mistralclient has a % tag too | 22:08 |
| clarkb | I'm going to work on my script to find these things then see if we can batch up the "fixes" | 22:09 |
| clarkb | http://mirror.ord.rax.openstack.org/ubuntu-ports/timestamp.txt shows that we've done a successful vos release | 22:10 |
| *** mriedem is now known as mriedem_afk | 22:18 | |
| clarkb | I have a script running to detect the cases of repo with no content | 22:20 |
| clarkb | three found so far including the one above | 22:20 |
| clarkb | (its not the fastest thing as it has to fetch every http page | 22:20 |
| fungi | clarkb: you ought to be able to find those locally on review.o.o with command-line git tools? | 22:24 |
| clarkb | fungi: if I'm looking for specific things like the % refs, but this can also happen if the repo is hidden | 22:24 |
| clarkb | I figured I'd do it against gitea and get the complete list of all unhappy repos then figure out why they are unhappy from there | 22:25 |
| fungi | ahh | 22:25 |
| clarkb | also the script for that is simple just not quick :) | 22:25 |
| clarkb | for X in `grep -- '- project:' gerrit/projects.yaml | sed -e 's/^. project: //'`; do curl -s https://opendev.org/$X | grep -q 'This repository does not contain any content.' && echo $X; done | 22:25 |
| fungi | maybe you can get it from the backend database for gitea? | 22:25 |
| clarkb | possibly | 22:25 |
| * fungi knows nothing of its schema | 22:25 | |
| clarkb | ya I dont wither | 22:25 |
| clarkb | *either | 22:25 |
| corvus | i'm not sure what to look for there | 22:27 |
| clarkb | no worries the script there is working | 22:27 |
| *** lifeless has quit IRC | 22:27 | |
| corvus | (especially since i think we've set the default branch on all projects) | 22:27 |
| clarkb | into the stackforge attic now | 22:27 |
| *** lifeless has joined #openstack-infra | 22:27 | |
| clarkb | http://paste.openstack.org/show/747752/ there are only three repos in this state | 22:28 |
| *** adriant has quit IRC | 22:29 | |
| clarkb | they all have the % tag name issue | 22:29 |
| clarkb | any objections to me applying the workaround we did earlier to those three repos? | 22:30 |
| fungi | are they all retired repos? | 22:30 |
| clarkb | yes | 22:30 |
| clarkb | (just finished checking that) | 22:30 |
| fungi | are they all deb-.* repos? | 22:30 |
| clarkb | yes | 22:30 |
| clarkb | (see the paste above) | 22:30 |
| fungi | nuke 'em | 22:30 |
| clarkb | will do | 22:31 |
| fungi | okay, this is interesting... looking at very recent failures it appears that tripleo-heat-templates is running tox-py35 rather than openstack-tox-py35 so is now failing because it wants to run on ubuntu-bionic by default | 22:33 |
| fungi | who do we normally reach out to about tripleo job problems? | 22:33 |
| clarkb | fungi: mwhahaha weshay zbr and rfolco | 22:35 |
| mwhahaha | probably stable branches | 22:35 |
| mwhahaha | we've moved to 36 in master | 22:35 |
| clarkb | openstack/deb-python-mistralclient already has a replacement tag with matching target ref so we only need to delete the bad tag there | 22:35 |
| fungi | neutron-lbaas-dashboard has a nodejs-npm-run-test which is returning retry_limit result because it wants https://deb.nodesource.com/node_4.x for bionic | 22:36 |
| fungi | so failing to set up npm in pre | 22:37 |
| mordred | fungi: I'd recommend that upgrades to a more recent node than 4.x | 22:37 |
| mordred | fungi: I believe 4 is no longer supported upstream | 22:38 |
| mordred | fungi: https://nodejs.org/en/about/releases/ | 22:38 |
| fungi | yeah, i mean that's probably why it's breaking | 22:38 |
| mordred | yeah | 22:38 |
| fungi | just pointing this out as something which may bite other projects as well | 22:38 |
| mordred | fungi: is that on a stable branch? | 22:39 |
| mordred | fungi: oh - doh. nevermindm. I can't computer | 22:40 |
| mordred | fungi: https://review.openstack.org/643177 Update node_version to 10 | 22:43 |
| mordred | fungi: we've also got some job templates in ozj that set node_version to 4 | 22:44 |
| mordred | this is making me think we should make a general effort to fix http://codesearch.openstack.org/?q=node_version%3A%204&i=nope&files=&repos= | 22:45 |
| fungi | likely | 22:46 |
| clarkb | #status log Added tag debian/1.6.0-2 to openstack/python-deb-oslotest as replacement for deleted tag debian/1%1.6.0-2 to workaround a gitea bug with % in ref names | 22:46 |
| openstackstatus | clarkb: finished logging | 22:46 |
| mordred | fungi: I'mma send a mailing list | 22:46 |
| clarkb | #status log Deleted tag debian/1%3.1.0-3 in openstack/deb-python-swiftclient to workaround gitea bug with % in ref names. There was already a debian/3.1.0-3 replacement tag pointing to the same ref. | 22:47 |
| openstackstatus | clarkb: finished logging | 22:47 |
| fungi | thanks mordred! | 22:47 |
| clarkb | #status log Deleted tag debian/1%2.1.2-1 in openstack/deb-python-mistralclient to workaround gitea bug with % in ref names. There was already a debian/2.1.2-1 replacement tag pointing to the same ref. | 22:47 |
| openstackstatus | clarkb: finished logging | 22:47 |
| mwhahaha | fungi: i don't see where tox-py35 is defined in tripleo so not sure where that's getting pulled in, we've got openstack-tox-py35 referenced | 22:48 |
| clarkb | If my script runs again and doesn't find any broken repos I'm going to start the gitea fixup playbook again | 22:49 |
| fungi | mwhahaha: i'll see if i can spot where it's coming in | 22:51 |
| fungi | zuul dashboard to the rescue | 22:51 |
| clarkb | I'm going to find some tea while waiting on my script | 22:52 |
| *** shardy has quit IRC | 22:52 | |
| fungi | mwhahaha: it's running for the https://review.openstack.org/600843 master branch change to tht | 22:53 |
| mwhahaha | yea i saw | 22:54 |
| mwhahaha | thats why i went looking | 22:54 |
| mwhahaha | but we don't pull it in in tripleo-ci or tripleo-heat-templates | 22:54 |
| mwhahaha | so it's coming in elsewhere | 22:54 |
| mwhahaha | wonder if it's from openstack-python35-jobs-no-constraints | 22:54 |
| mwhahaha | yea that's where it's coming from | 22:55 |
| mwhahaha | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n751 | 22:56 |
| mordred | fungi: we might need to put in a more systemic temporary workaround ... there are also a bunch of nodejs4- templates in ozj - and when you search for that, the carnage spreads a bit | 22:56 |
| mordred | fungi: I'll let someone smarter than me propose a course of action in reply to the mailing list though | 22:56 |
| fungi | mwhahaha: yeah, sorry, was digging for it as well and came to the same conclusion | 22:57 |
| mwhahaha | we can drop the job, though that job is used in openstack-infra/* stuff too | 22:57 |
| mwhahaha | so might need to fix tox-py35 | 22:57 |
| fungi | mwhahaha: well, there's no "fixing" tox-py35 since it's part of the zuul-jobs standard library, but we should be able to fix the templates in openstack-zuul-jobs themselves | 22:58 |
| *** shardy has joined #openstack-infra | 22:58 | |
| fungi | i'm looking at a possible patch now | 22:58 |
| mwhahaha | in the mean time i'll look at dropping it out of tripleo and switching to the other one | 22:59 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: Run tox-py35 on Xenial for no-constraints template https://review.openstack.org/643180 | 23:03 |
| fungi | mwhahaha: ^ | 23:03 |
| *** wolverineav has quit IRC | 23:03 | |
| mwhahaha | k thanks | 23:03 |
| fungi | config-core: ^ that's a fairly important bit i missed in the osj prep | 23:03 |
| fungi | quick review appreciated | 23:04 |
| fungi | fwiw i didn't see any other project-templates in osj calling tox-py35 directly | 23:04 |
| fungi | so hopefully that's all i overlooked there previously | 23:04 |
| *** cristoph has quit IRC | 23:04 | |
| clarkb | will look momentarily | 23:05 |
| *** slaweq has quit IRC | 23:05 | |
| *** ianychoi has joined #openstack-infra | 23:06 | |
| *** rlandy is now known as rlandy|bbl | 23:06 | |
| clarkb | gitea fixup is now running for hopefully the final time | 23:06 |
| *** rascasoft has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** wolverineav has joined #openstack-infra | 23:15 | |
| *** slaweq has quit IRC | 23:15 | |
| clarkb | corvus: mordred care to quickly review https://review.openstack.org/#/c/643180/ to fix up fungi's thing? | 23:16 |
| mordred | fungi: I think maybe we shoudl do the same thing for the nodejs4- project templates | 23:17 |
| fungi | mordred: please do | 23:18 |
| mordred | I mean - we should also get people off of nodejs4 - but it's possible that will take more than a second | 23:18 |
| fungi | that'll be a nearer-term fix at least | 23:18 |
| fungi | i've sent a recap to the ml with the several issues we've identified so far | 23:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Run nodejs4- jobs on ubuntu-xenial https://review.openstack.org/643182 | 23:20 |
| mordred | fungi: cool | 23:20 |
| *** wolverineav has quit IRC | 23:20 | |
| corvus | mordred: your message has a *folded* subject line! you almost never see those! | 23:22 |
| *** wolverineav has joined #openstack-infra | 23:22 | |
| corvus | (because early muas were terrible about handling them, and the habit stuck) | 23:22 |
| *** tosky has quit IRC | 23:23 | |
| corvus | clarkb: afsdb02 noticed afsdb01 was back online (a while ago): Wed Mar 13 21:46:06 2019 ubik:server 104.130.136.20 is back up: will be contacted through 104.130.136.20 | 23:26 |
| clarkb | corvus: is that from syslog? | 23:26 |
| corvus | so that's good | 23:26 |
| corvus | /var/log/openafs/VLLog | 23:27 |
| clarkb | thanks | 23:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run tox-py35 on Xenial for no-constraints template https://review.openstack.org/643180 | 23:27 |
| corvus | similar entry in PtLog | 23:27 |
| corvus | (that's "volume location" and "protection") | 23:27 |
| *** dave-mccowan has joined #openstack-infra | 23:28 | |
| corvus | (there are really two databases) | 23:28 |
| clarkb | ya I noticed that when backing up /var/lib/openafs | 23:28 |
| clarkb | and also bosserver is just a manager daemon it then forks off the appropriate sub processes | 23:28 |
| corvus | "basic overseer" iirc | 23:29 |
| clarkb | I'ev learned a lot about afs doing these upgrades | 23:29 |
| clarkb | and then bosconfig determines which of those subprocesses to run iirc | 23:30 |
| corvus | the ubik database/protocol was a gateway to PKD's short stories for me | 23:30 |
| *** lseki has quit IRC | 23:31 | |
| ianw | clarkb: yeah, it's almost like a system daemon that manages other daemons, daemon is hard to write so maybe we should just shorten it to "d" ... :) | 23:32 |
| clarkb | ianw: I think afs was probably invented before daemons existed | 23:33 |
| *** xek_ has quit IRC | 23:33 | |
| clarkb | ianw: a time of demogorgons and riding your bike down the street as a kid to save the world | 23:34 |
| clarkb | (stranger things is really good if you haven't seen it yet) | 23:34 |
| ianw | yeah, on the odd occasion i've had of going through the source looking for how an error message gets generated because it seems nobody else has ever seen that message (or at least posted about it somewhere indexable on the internet) it's been quite an archaeological tour | 23:34 |
| ianw | clarkb: yeah i watch that show and marvel at all the tupperware etc they've found. I'm like "we had that!". the americans too | 23:36 |
| *** mattw4 has quit IRC | 23:36 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run nodejs4- jobs on ubuntu-xenial https://review.openstack.org/643182 | 23:37 |
| *** dave-mccowan has quit IRC | 23:38 | |
| ianw | fungi: we got a bit sidetracked talking about https://review.openstack.org/#/c/639951/ for this bindep switching when afs went down the other day | 23:41 |
| ianw | i really don't like switching on the package manager, i just have that weird feeling that it may work but doesn't feel right | 23:41 |
| fungi | yeah, the rpm/dpkg nomenclature was meant more to identify distribution families based on packaging convention s | 23:44 |
| fungi | that distinction tended to represent a fair amount of the divergence in package naming between distros | 23:45 |
| fungi | i'm unconvinced that paradigm holds true with package managers themselves, at least any more than switching on distro and version does | 23:46 |
| *** wolverineav has quit IRC | 23:52 | |
| *** wolverineav has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!