| *** dayou_ has joined #openstack-infra | 00:01 | |
| *** dhill_ has joined #openstack-infra | 00:03 | |
| *** hamzy has joined #openstack-infra | 00:05 | |
| *** janki has quit IRC | 00:09 | |
| *** janki has joined #openstack-infra | 00:09 | |
| *** janki has quit IRC | 00:10 | |
| *** janki has joined #openstack-infra | 00:11 | |
| *** sthussey has quit IRC | 00:15 | |
| *** gyee has quit IRC | 00:19 | |
| *** jiapei has quit IRC | 00:24 | |
| *** jamesmcarthur has joined #openstack-infra | 00:26 | |
| ianw | hrm, this erw cloud seems to have the same problem as the linaro cloud, in that we end up with keystone telling us to use internal end-points for a range of operations (i think) | 00:36 |
|---|---|---|
| *** diablo_rojo has quit IRC | 00:36 | |
| fungi | could that be why openstackclient seemed to hate me when i was trying to test access to it? | 00:37 |
| ianw | fungi: no, i think that was a typo in your config :) | 00:38 |
| fungi | okay, that sounds FAR more likely ;) | 00:38 |
| ianw | however, openstackclient did hate me when i was doing something similar and i ended up with https://review.openstack.org/#/c/601485/ | 00:39 |
| ianw | and this time, i've noticed that the osc can log the password in plain per https://review.openstack.org/#/c/603528/ | 00:39 |
| ianw | so we're averaging a 100% chance of openstackclient issues when bringing up a new cloud so far :) | 00:40 |
| ianw | oh, i see in the latest change it's now arm64ci.cloud; i'm not using that, let me see if that makes any difference | 00:43 |
| ianw | it points at the same thing anyway | 00:44 |
| *** longkb has joined #openstack-infra | 00:46 | |
| *** jamesmcarthur has quit IRC | 00:59 | |
| *** harlowja has quit IRC | 01:02 | |
| ianw | ahhh, ok figured out the end-point thing. "interfaces: public" helps | 01:07 |
| *** diablo_rojo has joined #openstack-infra | 01:11 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** evrardjp has quit IRC | 01:12 | |
| *** slaweq has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #openstack-infra | 01:16 | |
| *** mrsoul has quit IRC | 01:29 | |
| *** anteaya has joined #openstack-infra | 01:30 | |
| *** studarus has joined #openstack-infra | 01:31 | |
| anteaya | folks may want to read my comments on this patch: https://review.openstack.org/#/c/602697 it is about a proposal to governance about social media use | 01:31 |
| anteaya | apart from the larger discussion | 01:31 |
| studarus | clarkb: instance 4ee38e14-1181-40b1-bc00-ef479514f21b has two private IP addresses assigned... I'll do some more investigating. End result is we run out of ports on that subnet... | 01:32 |
| anteaya | there is a very helpful comment from a user from korea and I offered some thoughts, others may want to offer theirs | 01:32 |
| anteaya | also that user may show up in channel and have some questions as I have suggested they do just that | 01:32 |
| *** hongbin has joined #openstack-infra | 01:37 | |
| *** studarus has quit IRC | 01:38 | |
| *** diablo_rojo has quit IRC | 01:43 | |
| *** ykarel|away has joined #openstack-infra | 01:52 | |
| *** imacdonn has joined #openstack-infra | 01:54 | |
| imacdonn | in case anyone's around .... something seems unwell with the gate ... 603194,1 has been in there for 10 hours, and seems to be in a loop .. it gets almost finished, then starts over | 01:56 |
| *** dpawlik has joined #openstack-infra | 01:56 | |
| *** anteaya has quit IRC | 01:58 | |
| *** dpawlik has quit IRC | 02:00 | |
| openstackgerrit | Merged openstack-infra/git-review master: Always print failure case when testing remotes https://review.openstack.org/602767 | 02:00 |
| *** anteaya has joined #openstack-infra | 02:06 | |
| clarkb | imacdonn: I think that us because the changes ahead of it in the gate keep failing causing it to start over | 02:09 |
| clarkb | not ideal, but zuul is doing what we have asked of it | 02:09 |
| *** jamesmcarthur has quit IRC | 02:12 | |
| *** armax has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** anteaya has quit IRC | 02:24 | |
| *** linshuicheng[m] is now known as linshuicheng[m]1 | 02:25 | |
| *** ijw has joined #openstack-infra | 02:31 | |
| *** ijw has quit IRC | 02:36 | |
| *** ykarel|away has quit IRC | 02:43 | |
| imacdonn | clarkb: OK... hope it makes it through eventually ... kinda seems like zuul is only creating more work for itself by timing out (?) and repeating over and over | 02:44 |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** _ari_ has quit IRC | 02:45 | |
| *** rascasoft has quit IRC | 02:45 | |
| *** apetrich has quit IRC | 02:51 | |
| *** armax has joined #openstack-infra | 02:52 | |
| *** vivsoni has joined #openstack-infra | 02:56 | |
| *** armax has quit IRC | 02:56 | |
| *** jistr has quit IRC | 03:00 | |
| *** jistr has joined #openstack-infra | 03:00 | |
| prometheanfire | clarkb: looks like app-portage/gentoolkit is expected to be installed http://logs.openstack.org/46/602446/4/check/openstack-infra-base-integration-gentoo-17-0-systemd/7fd4e0c/job-output.txt.gz#_2018-09-18_19_30_14_106743 | 03:01 |
| prometheanfire | clarkb: where is best to add that? | 03:02 |
| prometheanfire | probably in one of the pre-elements or something | 03:04 |
| *** ramishra has joined #openstack-infra | 03:08 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: Install gentoolkit on Gentoo https://review.openstack.org/603544 | 03:08 |
| prometheanfire | clarkb: well, let me know if it's the right place/element ^ | 03:08 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** cgoncalves|pto has quit IRC | 03:16 | |
| *** slaweq has quit IRC | 03:16 | |
| *** cgoncalves has joined #openstack-infra | 03:17 | |
| *** eernst has quit IRC | 03:20 | |
| ianw | fungi / gary_perkins : sent an email with some bits on the new cloud, thanks. LMN thoughts on projects etc | 03:34 |
| *** jamesmcarthur has quit IRC | 03:34 | |
| *** vivsoni has quit IRC | 03:37 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:50 | |
| *** udesale has joined #openstack-infra | 03:52 | |
| *** ykarel|away has joined #openstack-infra | 03:53 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** vivsoni has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** vivsoni has quit IRC | 04:06 | |
| *** vivsoni has joined #openstack-infra | 04:06 | |
| *** yamamoto has joined #openstack-infra | 04:07 | |
| *** rfolco has quit IRC | 04:08 | |
| *** jamesmcarthur has joined #openstack-infra | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:13 | |
| *** jaosorior_ is now known as jaosorior | 04:13 | |
| *** jamesmcarthur has joined #openstack-infra | 04:23 | |
| *** jamesmcarthur has quit IRC | 04:27 | |
| *** diablo_rojo has joined #openstack-infra | 04:28 | |
| *** bobh has quit IRC | 04:31 | |
| *** harlowja has joined #openstack-infra | 04:32 | |
| *** rkukura has quit IRC | 04:35 | |
| *** rkukura has joined #openstack-infra | 04:35 | |
| *** bobh has joined #openstack-infra | 04:37 | |
| *** bobh has quit IRC | 04:42 | |
| *** bobh has joined #openstack-infra | 04:51 | |
| *** roman_g has quit IRC | 04:52 | |
| *** bobh has quit IRC | 04:55 | |
| openstackgerrit | Ilya Etingof proposed openstack-infra/git-review master: Improve exit code implementation https://review.openstack.org/480267 | 04:57 |
| *** hamzy_ has joined #openstack-infra | 05:01 | |
| *** hamzy has quit IRC | 05:02 | |
| *** ykarel|away has quit IRC | 05:04 | |
| *** hongbin has quit IRC | 05:06 | |
| *** diablo_rojo has quit IRC | 05:07 | |
| *** hamzy has joined #openstack-infra | 05:09 | |
| *** sshnaidm has joined #openstack-infra | 05:10 | |
| *** hamzy_ has quit IRC | 05:11 | |
| *** sshnaidm has quit IRC | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** hamzy has quit IRC | 05:17 | |
| *** hamzy has joined #openstack-infra | 05:18 | |
| *** hamzy has quit IRC | 05:23 | |
| *** hamzy has joined #openstack-infra | 05:24 | |
| *** rcernin has quit IRC | 05:30 | |
| *** rcernin has joined #openstack-infra | 05:30 | |
| *** quique|rover|off is now known as quiquell|rover | 05:33 | |
| ianw | niedbalski: it too me *far* too long to realise, but "identinty_interface: public" doesn't actually do anything in clouds.yaml ... it's on "interface:" that works | 05:37 |
| ianw | only | 05:37 |
| ianw | however, it is "identity_api_version:" for the api version, and you can prefix like "compute_interface" if you're calling certain parts of the client directly from code | 05:39 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable zuul_return to update Ansible inventory https://review.openstack.org/590092 | 05:44 |
| *** ykarel|away has joined #openstack-infra | 05:47 | |
| *** jtomasek has quit IRC | 06:00 | |
| *** jtomasek has joined #openstack-infra | 06:01 | |
| *** rcernin_ has joined #openstack-infra | 06:04 | |
| *** jtomasek has quit IRC | 06:06 | |
| *** rcernin has quit IRC | 06:06 | |
| *** holser_ has joined #openstack-infra | 06:12 | |
| *** janki has quit IRC | 06:17 | |
| *** holser_ has quit IRC | 06:18 | |
| *** slaweq has joined #openstack-infra | 06:18 | |
| *** ykarel|away is now known as ykarel | 06:21 | |
| *** holser_ has joined #openstack-infra | 06:21 | |
| *** slaweq has quit IRC | 06:23 | |
| *** harlowja has quit IRC | 06:24 | |
| *** dpawlik has joined #openstack-infra | 06:27 | |
| *** dpawlik has quit IRC | 06:28 | |
| *** dpawlik_ has joined #openstack-infra | 06:28 | |
| *** dpawlik_ has quit IRC | 06:30 | |
| *** dpawlik has joined #openstack-infra | 06:30 | |
| *** dpawlik has quit IRC | 06:31 | |
| *** dpawlik_ has joined #openstack-infra | 06:31 | |
| *** rcernin has joined #openstack-infra | 06:34 | |
| *** rcernin_ has quit IRC | 06:36 | |
| *** jtomasek has joined #openstack-infra | 06:38 | |
| *** chkumar|off is now known as chkumar|ruck | 06:53 | |
| *** slaweq has joined #openstack-infra | 06:55 | |
| *** janki has joined #openstack-infra | 06:55 | |
| *** apetrich has joined #openstack-infra | 06:57 | |
| *** slaweq has quit IRC | 07:00 | |
| *** rcernin has quit IRC | 07:06 | |
| *** olivierb has joined #openstack-infra | 07:09 | |
| *** hashar has joined #openstack-infra | 07:09 | |
| *** jamesmcarthur has joined #openstack-infra | 07:13 | |
| *** slaweq has joined #openstack-infra | 07:16 | |
| *** jamesmcarthur has quit IRC | 07:17 | |
| *** ykarel is now known as ykarel|lunch | 07:25 | |
| egonzalez | hi, EPEL mirror still missing some packages which are on public epel https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/u/uwsgi-plugin-python2-2.0.17.1-1.el7.x86_64.rpm | 07:28 |
| egonzalez | well, package is actually there http://mirror.mtl01.inap.openstack.org/epel/7/x86_64/Packages/u/uwsgi-plugin-python2-2.0.17.1-1.el7.x86_64.rpm | 07:29 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Cleanup of leaked resource for static driver https://review.openstack.org/600084 | 07:30 |
| egonzalez | but building docker images in queens branch fails because the missing package, not in rocky or master | 07:30 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Implement liveness check for static nodes https://review.openstack.org/601513 | 07:30 |
| egonzalez | any idea why can this be happening? | 07:30 |
| egonzalez | INFO:kolla.image.build.barbican-base: * epel: mirror.mtl01.inap.openstack.org | 07:32 |
| egonzalez | INFO:kolla.image.build.barbican-base:No package uwsgi-plugin-python available. | 07:32 |
| ianw | egonzalez: is it an upstream issue? I'm not seeing any issues with the epel mirroring process | 07:34 |
| ianw | can you link to the full logs? | 07:35 |
| *** e0ne has joined #openstack-infra | 07:36 | |
| *** tosky has joined #openstack-infra | 07:37 | |
| *** jtomasek has quit IRC | 07:37 | |
| *** Gorian has quit IRC | 07:37 | |
| *** jesusaur has quit IRC | 07:39 | |
| *** pcrews has quit IRC | 07:39 | |
| *** jtomasek has joined #openstack-infra | 07:39 | |
| *** ginopc has joined #openstack-infra | 07:40 | |
| *** jesusaur has joined #openstack-infra | 07:44 | |
| *** e0ne has quit IRC | 07:48 | |
| strigazi | hello AJaeger, after this you proposed in magnum, http://git.openstack.org/cgit/openstack/magnum/commit/?id=4a1a4be0d315dbce44fd569b491c989a403017c0 the cover job is voting. How can we make it non-voting? | 07:51 |
| egonzalez | ianw yep, the stable/queens jobs are failing due missing uswgi-plugin-python package missing http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/kolla/stable/queens/kolla-publish-centos-binary/c3a643e/logs/build/000_FAILED_barbican-api.txt.gz | 07:56 |
| ianw | egonzalez : hrm, i think maybe that's a red herring | 07:57 |
| ianw | before that | 07:57 |
| ianw | Timeout on https://copr-be.cloud.fedoraproject.org/results/iwienand/zookeeper-el7/epel-7-x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds') | 07:57 |
| ianw | perhaps copr is having downtime? | 07:57 |
| ianw | egonzalez: actually, maybe again ... Determining fastest mirrors | 08:01 |
| ianw | INFO:kolla.common.utils.barbican-api: * epel: fedora-epel.mirror.iweb.com | 08:01 |
| ianw | i don't think it's using our mirror | 08:01 |
| egonzalez | ianw the last attempt tries to use openstack mirrors INFO:kolla.common.utils.barbican-api:Determining fastest mirrors | 08:02 |
| egonzalez | INFO:kolla.common.utils.barbican-api: * epel: mirror.mtl01.inap.openstack.org | 08:02 |
| egonzalez | ianw btw, this is the replace we use in gates for the mirrors https://github.com/openstack/kolla/blob/d609c318bf374217c5d2e40a9e51dd565581333d/tests/templates/template_overrides.j2#L40 | 08:06 |
| *** Emine has joined #openstack-infra | 08:06 | |
| egonzalez | hrm, may be the fedora url changed, http://download.fedoraproject.org/pub to http://dl.fedoraproject.org/pub | 08:08 |
| *** jpich has joined #openstack-infra | 08:08 | |
| egonzalez | in master is getting the package from delorean, thats why is not failing | 08:14 |
| *** rossella_s has quit IRC | 08:14 | |
| *** rossella_s has joined #openstack-infra | 08:15 | |
| *** roman_g has joined #openstack-infra | 08:17 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 08:20 |
| AJaeger | strigazi: remove the cover template, move it to check queue with "voting: false" to it. | 08:21 |
| strigazi | AJaeger: thanks | 08:22 |
| AJaeger | strigazi: why don't you want it voting? Just curious | 08:22 |
| AJaeger | strigazi: happy to review a change - but no time right now to do it myself | 08:22 |
| strigazi | AJaeger It is buggy, it has false negative, until we fix it, we have it non-voting | 08:22 |
| AJaeger | I see. You can also leave the template in in that case | 08:22 |
| strigazi | I didn't get the last comment | 08:23 |
| AJaeger | strigazi: let me do it quickly for you... | 08:23 |
| strigazi | AJaeger: https://review.openstack.org/603001 | 08:24 |
| AJaeger | https://review.openstack.org/603594 - I can abandon again ;) | 08:25 |
| *** e0ne has joined #openstack-infra | 08:25 | |
| AJaeger | strigazi: my change is correct - your call on how to continue... | 08:25 |
| * AJaeger needs to step out again | 08:26 | |
| *** e0ne has quit IRC | 08:26 | |
| strigazi | AJaeger: thanks for your time, I really really apreciate it. | 08:26 |
| *** jpena|off is now known as jpena | 08:31 | |
| *** e0ne has joined #openstack-infra | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:41 | |
| *** alexchadin has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:49 | |
| *** owalsh has quit IRC | 08:51 | |
| *** owalsh has joined #openstack-infra | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| *** e0ne has quit IRC | 09:02 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul master: Assure that status tooltip is displayed on entire row https://review.openstack.org/603504 | 09:05 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: opensuse-mirror: Switch to US mirror for OBS repositories https://review.openstack.org/603610 | 09:05 |
| hwoarang | infra-root: ^^ could we get this in please? it's impacting jobs again :( | 09:06 |
| *** gfidente has joined #openstack-infra | 09:12 | |
| *** dpawlik_ has quit IRC | 09:13 | |
| *** dpawlik has joined #openstack-infra | 09:13 | |
| *** markvoelker has quit IRC | 09:25 | |
| *** elod has quit IRC | 09:26 | |
| *** elod has joined #openstack-infra | 09:27 | |
| *** dangtrinhnt has joined #openstack-infra | 09:41 | |
| *** owalsh has quit IRC | 09:46 | |
| *** owalsh has joined #openstack-infra | 09:52 | |
| ssbarnea | ianw: it took some time to add unittests but it was possible, see https://review.openstack.org/#/c/583535/ -- tripled the efforth but now we can avoid regressions. | 09:52 |
| *** ijw has joined #openstack-infra | 09:54 | |
| mordred | slaweq: morning! | 09:58 |
| *** ijw has quit IRC | 09:58 | |
| slaweq | mordred: hi | 09:58 |
| slaweq | isn't it very early for You? :) | 09:58 |
| mordred | slaweq: yes, yes it is :) | 09:59 |
| mordred | slaweq: but - maybe that's a good thing for helping you track down the neutron thing? | 09:59 |
| slaweq | mordred: thx a lot | 10:00 |
| slaweq | mordred: basically I have two things two debug, one, more important is failing neutron-grenade-dvr-multinode job | 10:00 |
| slaweq | but biggest problem with this one is that when I logged in to node, everything worked fine :/ | 10:01 |
| slaweq | so I'm now trying to debug it by adding some additional logs and running it in gate | 10:01 |
| slaweq | maybe I will figure out what's going on there | 10:01 |
| mordred | slaweq: ugh, that sounds like fun | 10:02 |
| slaweq | mordred: yup :) | 10:02 |
| slaweq | mordred: but maybe You can help me with this second thing | 10:02 |
| mordred | I hope so | 10:02 |
| slaweq | mordred: I still have no idea why in job neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 (https://review.openstack.org/#/c/578796/) all FIP from subnode-2 are not reachable | 10:03 |
| slaweq | I suspect that there is some difference between this "new" multinode setup and old legacy job | 10:03 |
| slaweq | can You maybe set for me on autohold both those jobs? I would recheck my patch and then I would be maybe able to log to both of them and compare config of nodes | 10:04 |
| mordred | slaweq: sure! also - we should pull in clarkb when he gets up on the floating ip being reachable thing - he is the master of the networking setup for multinode jobs | 10:05 |
| mordred | slaweq: autoholds are now set | 10:05 |
| slaweq | mordred: good to know that, so I will compare those configs and then I will get back to clarkb if I will need any help with this and if this will be really nodes' networking setup | 10:05 |
| slaweq | thx a lot | 10:06 |
| slaweq | is it on hold for specifi project or for any project? | 10:06 |
| slaweq | I mean, will it be holded if will fails on neutron patch instead of neutron-tempest-plugin? | 10:06 |
| mordred | oh - I did it for the same one as last time - so ... | 10:06 |
| mordred | yeah, openstack/neutron-tempest-plugin for neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 | 10:07 |
| mordred | although I could set it for a different project if you prefer | 10:07 |
| slaweq | that is what I want :) | 10:07 |
| mordred | sweet | 10:07 |
| slaweq | thx a lot | 10:07 |
| slaweq | if I will have such failed job I will come back to You :) | 10:07 |
| dangtrinhnt | Hi openstack-infra team. Sorry for interrupting your conversation. I'm trying to add myself as the channel's operator of the #openstack-searchlight channel (I'm the PTL) but helpless. And I cannot contact the last PTL. The open-infra docs says something about setting mask as full_mask but I don't think it's it. | 10:11 |
| dangtrinhnt | It would be great if someone can give me a hint. Thanks. | 10:12 |
| ianw | dangtrinhnt: are you messaging chanserv? | 10:20 |
| dangtrinhnt | Pardon me. I don't understand your question. :) I'm not the operator of #openstack-searchlight because I just joint last month. | 10:29 |
| dangtrinhnt | oh, looks like you grant the rights for me. Many thanks. | 10:29 |
| dangtrinhnt | ianw | 10:29 |
| *** markvoelker has joined #openstack-infra | 10:30 | |
| openstackgerrit | Merged openstack-infra/puppet-asterisk master: Ensure asterisk refresh happens last https://review.openstack.org/601749 | 10:31 |
| *** aidin has joined #openstack-infra | 10:34 | |
| *** longkb has quit IRC | 10:34 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for pbx.o.o https://review.openstack.org/601837 | 10:37 |
| *** longkb has joined #openstack-infra | 10:38 | |
| *** longkb has quit IRC | 10:39 | |
| *** longkb has joined #openstack-infra | 10:40 | |
| *** longkb has quit IRC | 10:42 | |
| *** e0ne has joined #openstack-infra | 10:47 | |
| *** aidin has left #openstack-infra | 10:50 | |
| *** markvoelker has quit IRC | 11:00 | |
| *** panda has joined #openstack-infra | 11:00 | |
| *** priteau has joined #openstack-infra | 11:16 | |
| *** imacdonn has quit IRC | 11:19 | |
| *** imacdonn has joined #openstack-infra | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| *** pbourke has quit IRC | 11:22 | |
| *** pbourke has joined #openstack-infra | 11:24 | |
| *** dhill_ has quit IRC | 11:25 | |
| *** sambetts|afk is now known as sambetts | 11:35 | |
| *** ssbarnea has quit IRC | 11:39 | |
| *** owalsh has quit IRC | 11:42 | |
| *** rh-jelabarre has joined #openstack-infra | 11:46 | |
| *** _Cyclone_ has quit IRC | 11:48 | |
| *** ansmith has quit IRC | 11:49 | |
| *** _Cyclone_ has joined #openstack-infra | 11:52 | |
| quiquell|rover | Hello | 11:52 |
| quiquell|rover | dangtrinhnt, mordred: Do you know what happends with zuul_changes for this https://review.openstack.org/#/c/594145/ ? | 11:53 |
| quiquell|rover | It has a Depends-On that point to multiple reviews | 11:53 |
| quiquell|rover | looks like zuul_changes just show the puppet-tripleo ZUUL_CHANGES=openstack/puppet-tripleo:stable/queens:refs/changes/45/594145/3 | 11:54 |
| AJaeger | infra-root, OVH mainteance should be over in 5 minutes and I'll then approve https://review.openstack.org/#/c/603174/ to get back the cloud. Anybody else to review? | 11:55 |
| *** ssbarnea has joined #openstack-infra | 11:56 | |
| *** aperevalov has quit IRC | 11:56 | |
| *** udesale has quit IRC | 11:56 | |
| *** markvoelker has joined #openstack-infra | 11:57 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/project-config master: Allow push and push merge commit for tripleo-quickstart https://review.openstack.org/602377 | 11:59 |
| *** ijw has joined #openstack-infra | 12:02 | |
| mordred | quiquell|rover: I'm not sure about ZUUL_CHANGES (that's a legacy compat thing and I'm not 100% sure of the interactions), but I highly recommend switching the depends-on to use the url form instead of the changeid form - because regardless of what goes into zuul_changes, zuul will not merge that patch until all three matching changes have landed | 12:04 |
| mordred | heh. seems like you got that taken care of | 12:05 |
| mordred | AJaeger: I tossed a +2 on there | 12:05 |
| *** ijw has quit IRC | 12:07 | |
| AJaeger | thanks, mordred | 12:08 |
| quiquell|rover | mordred: thanks, we have change to full url | 12:11 |
| quiquell|rover | mordred: we are still depending on legacy ZUUL_CHANGES | 12:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC https://review.openstack.org/603174 | 12:12 |
| *** dotplus has left #openstack-infra | 12:14 | |
| *** rfolco has joined #openstack-infra | 12:16 | |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| *** e0ne has quit IRC | 12:20 | |
| *** hashar is now known as hasharAway | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** ijw has joined #openstack-infra | 12:26 | |
| *** ijw has quit IRC | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:31 | |
| *** markvoelker has quit IRC | 12:31 | |
| *** quiquell|rover is now known as quique|rover|lch | 12:33 | |
| *** kgiusti has joined #openstack-infra | 12:42 | |
| openstackgerrit | Merged openstack-infra/system-config master: modules: opensuse-mirror: Switch to US mirror for OBS repositories https://review.openstack.org/603610 | 12:44 |
| AJaeger | infra-root, can you check the graphs for OVH BHS1? Is everything looking fine in that cloud? I see 10 launch errors/min and wonder whether thta is fine | 12:47 |
| dpawlik | AJaeger: We are still upgrading the infra | 12:49 |
| dpawlik | AJaeger: it is possible that some part of compute hosts has been disabled. Please wait one/two hours | 12:50 |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** dhill_ has joined #openstack-infra | 12:52 | |
| *** alexchadin has quit IRC | 12:53 | |
| AJaeger | dpawlik: oh - should we swithc them off again? | 12:53 |
| dpawlik | AJaeger: if you can just wait some time :) | 12:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC" https://review.openstack.org/603741 | 12:54 |
| *** alexchadin has joined #openstack-infra | 12:54 | |
| dpawlik | AJaeger: or maybe I will ask team responsible for upgrade if they can upgrade OS aggregation | 12:54 |
| AJaeger | dpawlik: just tell us when you're ready, please | 12:54 |
| AJaeger | infra-root, can we promote 603741 ? | 12:55 |
| AJaeger | mordred: ^ | 12:55 |
| EmilienM | hello infra, can someone approve https://review.openstack.org/#/c/602869/ ? thanks | 12:55 |
| *** trown|outtypewww is now known as trown|brb | 12:55 | |
| *** trown|brb is now known as trown | 12:56 | |
| *** vivsoni has quit IRC | 12:56 | |
| *** vivsoni has joined #openstack-infra | 12:56 | |
| dpawlik | AJaeger: you are a "prio" :) | 12:57 |
| AJaeger | dpawlik: thanks ;) Just takes time right now to switch them off again, I don#t have the permissions to help immediately. | 12:58 |
| *** kukacz_ is now known as kukacz | 13:00 | |
| *** ansmith has joined #openstack-infra | 13:04 | |
| *** holser_ has quit IRC | 13:04 | |
| *** holser__ has joined #openstack-infra | 13:04 | |
| AJaeger | infra-root, or change the value in nodepool for OVH BHS1 directly? | 13:09 |
| mordred | AJaeger: looking | 13:09 |
| *** alexchadin has quit IRC | 13:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC" https://review.openstack.org/603741 | 13:11 |
| AJaeger | thanks, mordred | 13:11 |
| AJaeger | dpawlik: ^ | 13:12 |
| mordred | AJaeger: :53 | 13:12 |
| mordred | gah | 13:12 |
| mordred | AJaeger: I also edited the file on nl04 directly | 13:12 |
| AJaeger | mordred: thanks | 13:15 |
| AJaeger | dpawlik: ok, we should not launch anything anymore... | 13:15 |
| *** alexchadin has joined #openstack-infra | 13:15 | |
| *** janki has quit IRC | 13:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Revert "Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC"" https://review.openstack.org/603766 | 13:18 |
| *** jamesmcarthur has quit IRC | 13:18 | |
| cmurphy | love the triple revert | 13:18 |
| AJaeger | infra-root, please take over once OVH is ready ^ - no further time | 13:18 |
| *** jamesmcarthur has joined #openstack-infra | 13:19 | |
| *** sthussey has joined #openstack-infra | 13:20 | |
| *** quique|rover|lch is now known as quiquell|rover | 13:21 | |
| *** mriedem has joined #openstack-infra | 13:21 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add openstack-python36 job to Neutron Grafana dashboard https://review.openstack.org/595573 | 13:28 |
| *** tpsilva has joined #openstack-infra | 13:36 | |
| *** udesale has joined #openstack-infra | 13:38 | |
| *** bobh has joined #openstack-infra | 13:46 | |
| *** jamesmcarthur has quit IRC | 13:49 | |
| *** ginopc has quit IRC | 13:50 | |
| *** jamesmcarthur has joined #openstack-infra | 13:50 | |
| *** bobh has quit IRC | 13:51 | |
| *** rascasoft has joined #openstack-infra | 13:51 | |
| *** ginopc has joined #openstack-infra | 13:52 | |
| *** ginopc has quit IRC | 13:56 | |
| *** bobh has joined #openstack-infra | 13:58 | |
| *** janki has joined #openstack-infra | 13:59 | |
| *** bobh has quit IRC | 14:02 | |
| *** janki has quit IRC | 14:04 | |
| *** ykarel is now known as ykarel|away | 14:04 | |
| *** ginopc has joined #openstack-infra | 14:04 | |
| slaweq | mordred: hi | 14:06 |
| slaweq | mordred: one of my jobs just failed: http://logs.openstack.org/96/578796/18/check/neutron-tempest-plugin-dvr-multinode-scenario-zuulv3/e1620d0/ | 14:06 |
| slaweq | can You check those nodes and add my ssh key to them? | 14:06 |
| dpawlik | slaweq: so sad :P | 14:06 |
| slaweq | dpawlik: why? I was waiting for that since morning :) | 14:07 |
| *** aidin has joined #openstack-infra | 14:07 | |
| dpawlik | slaweq: oh | 14:07 |
| *** jtomasek has quit IRC | 14:08 | |
| *** bobh has joined #openstack-infra | 14:08 | |
| mordred | slaweq: ok - looks like both jobs failed actually- so 198.72.124.232 and 198.72.124.237 are neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 | 14:10 |
| mordred | slaweq: and 23.253.201.43 and 23.253.213.20 are neutron-grenade-dvr-multinode | 14:11 |
| mordred | slaweq: oh - wait - I got those backwards | 14:11 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: Install gentoolkit on Gentoo https://review.openstack.org/603544 | 14:11 |
| slaweq | mordred: thx a lot | 14:11 |
| mordred | slaweq: 23. are neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 - 198. are neutron-grenade-dvr-multinode | 14:11 |
| slaweq | ok, great :) | 14:11 |
| slaweq | thx | 14:11 |
| mordred | sure thing! | 14:11 |
| *** bobh has quit IRC | 14:12 | |
| efried | Good UGT morning all. Random question for gerrit query mavens: Are wildcards/regexes of any kind supported? E.g. if I want to search by topic:bug/.* kind of thing? | 14:15 |
| cmurphy | efried: it works best if you double quote it and use ^ and $ | 14:17 |
| cmurphy | topic:"^bug/.*$" | 14:17 |
| efried | hmph, snot working for me. | 14:18 |
| *** bobh has joined #openstack-infra | 14:19 | |
| cmurphy | hmm maybe it doesn't work for topics | 14:19 |
| efried | boo. Okay, thanks cmurphy. | 14:20 |
| slaweq | mordred: there is 2 small problems with neutron-grenade-dvr-multinode | 14:22 |
| slaweq | mordred: 1. it's upgrade from ocata to pike and that is not what I was looking for :/ | 14:22 |
| slaweq | mordred: 2. in fact I wanted today to set autohold on neutron-tempest-plugin-dvr-multinode-scenario job in neutron-tempest-plugin instead of this grenade job :D | 14:22 |
| slaweq | because I wanted to compare it with setup of neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 which should be the same but isn't for some reason | 14:23 |
| *** alexchadin has quit IRC | 14:25 | |
| *** bobh has quit IRC | 14:27 | |
| clarkb | slaweq: mordred dvr requires we use the test env overlay to route FIPs between nodes as they can be terminated on each node when using dvr | 14:29 |
| clarkb | this isnt necessary without dvr because all FIPs are terminated on the controller and we run tempest there so all IPs are local | 14:29 |
| *** alexchadin has joined #openstack-infra | 14:30 | |
| clarkb | I would double check you have the test env overlay in place for br-ex on the dvr job | 14:30 |
| mordred | slaweq: well poo. so should I set an autohold for neutron-tempest-plugin-dvr-multinode-scenario on neutron-tempest-plugin? | 14:31 |
| *** bobh has joined #openstack-infra | 14:32 | |
| clarkb | we have ascii art about this somewhere too | 14:33 |
| clarkb | but the issue is routing arbitrary FIP range in arbitrary cloud netowkring | 14:33 |
| smcginnis | fungi: You had mentioned in a thread about making our cgit interface nicer. Has anyone looked at gogs.io? | 14:33 |
| mordred | smcginnis: yah. I looked at it a while back | 14:34 |
| clarkb | we do that with a vxlan overlay between test nodes which gives us direct attached routes to the larger overlay range thrn neutron assigns FIPs out of a subset | 14:34 |
| mordred | smcginnis: I like it conceptually, but I haven't gotten far enough to make any formal suggestions | 14:34 |
| smcginnis | mordred: Just a quick local trial, I kind of like it. | 14:34 |
| mordred | smcginnis: luckily it DOES allow completely disabling pull requests, issues, wikis, etc | 14:34 |
| smcginnis | mordred: And it might make it easier for those more familiar with github. | 14:35 |
| smcginnis | ++ | 14:35 |
| clarkb | gogs has a fork too also they were owned on github iirc | 14:35 |
| clarkb | (and dont self host) | 14:35 |
| smcginnis | I was wondering though, I wonder if we could somehow bridget that pull request interface into being able to submit reviews. | 14:35 |
| smcginnis | Another way to make it easier for folks more familiar with github. | 14:35 |
| mordred | yah - that is a thing that is a much harder thing | 14:35 |
| *** gouthamr_ is now known as gouthamr | 14:35 | |
| mordred | because we'd need to make an entire UI interface that used gerrit as a backend | 14:35 |
| smcginnis | Yeah, that would probably require a fair amount of noodling and work. | 14:36 |
| mordred | yup | 14:36 |
| clarkb | gitea | 14:36 |
| *** bobh has quit IRC | 14:36 | |
| smcginnis | clarkb: Oh, that looks nice too. Very similar. | 14:36 |
| mordred | clarkb: https://notabug.org/hp/gogs/ is the fork I was looking at | 14:36 |
| mordred | but any of them would be fine | 14:36 |
| clarkb | gitea is a community managed fork of gogs because gogs went idle | 14:37 |
| smcginnis | cgit is a litte... aged. | 14:37 |
| * prometheanfire likes cgit :P | 14:37 | |
| clarkb | But my big issue with all of the is non self host | 14:37 |
| smcginnis | prometheanfire is a little... aged. | 14:37 |
| smcginnis | :P | 14:37 |
| clarkb | if they arent good enough for their authors... | 14:37 |
| *** udesale has quit IRC | 14:37 | |
| smcginnis | clarkb: Hah, I chuckled when I noticed that too. | 14:37 |
| mordred | clarkb: notabug self-hosts | 14:38 |
| prometheanfire | smcginnis: iirc we have a dev for it, so are likely to stay on it | 14:38 |
| prometheanfire | really just depends on what you want though | 14:38 |
| smcginnis | Yeah, it totally works. Just many people end up browsing the github mirrors instead since it's so darn purdy. | 14:39 |
| clarkb | mordred: ah neat | 14:39 |
| prometheanfire | I'm one of those, but more because of muscle memory | 14:39 |
| mordred | I like the simplicity of cgit and the fact that it doesn't require a database or anything lke that to run. however, I will admit that github/gogs are a nicer browsing experience for humans | 14:39 |
| AJaeger | clarkb: did you see scrollback about OVH? | 14:39 |
| clarkb | AJaeger: ya looks like dpawlik will give us tue go ahead? | 14:40 |
| mordred | and with a legit open source choice in gogs that we coudl use to make git.openstack.org more prettier, I don't think it's a bad idea to explore it | 14:40 |
| *** dayou_ has quit IRC | 14:40 | |
| prometheanfire | switching to blame view, and going back in history is nice | 14:40 |
| AJaeger | clarkb: yes. Could you keep this on your radar, please? I'm busy right now... | 14:40 |
| prometheanfire | not sure if cgit has the compare function that github has (which I use a ton) | 14:40 |
| smcginnis | I would be willing to help however I can if there is enough interest to make a change. | 14:40 |
| smcginnis | prometheanfire: ++ | 14:40 |
| *** bobh has joined #openstack-infra | 14:40 | |
| clarkb | mordred: smcginnis github UI has some huge flaws for browsing though. md/rst are always rendered or raw so no line linking. Also line width is like 80 chars so glhf browsing wide code | 14:40 |
| clarkb | AJaeger: yup | 14:40 |
| mordred | smcginnis: most of the effort will likely be around integrating with project creation workflow | 14:41 |
| smcginnis | clarkb: Yeah, that's about the only thing I like better in cgit. :D | 14:41 |
| mordred | smcginnis, clarkb but with the opendev stuff coming up - perhaps it's worthy of a spec | 14:41 |
| clarkb | mordred: if we are picking priorities gerrit upgrade and actual opendev work is probably far ahead of this | 14:42 |
| mordred | I would be happy to volunteer to write one and then dump work on smcginnis :) | 14:42 |
| mordred | clarkb: oh - totally | 14:42 |
| smcginnis | :) | 14:42 |
| slaweq | clarkb: thx for tips, basically I'm using tempest-multinode-full job as parent for my job | 14:42 |
| slaweq | clarkb: so I was thinking that it will configure everything for me just fine | 14:43 |
| *** quiquell|rover is now known as quique|rover|off | 14:43 | |
| mordred | but if we had some stuff written down, someone (such as smcginnis) could reasonably work on some of the pre-reqs while we work through the other priorities | 14:43 |
| clarkb | smcginnis: mordred for code submission I think bridging PRs and changes in gerrit is likely to only lead to pain | 14:43 |
| prometheanfire | smcginnis: if you have an account this should work | 14:43 |
| clarkb | now you have to understand two workflows a d how they go together | 14:43 |
| mordred | clarkb: yah - I do not desire to do that | 14:43 |
| prometheanfire | https://try.gogs.io/gogs/gogs/compare/v0.11.53...v0.11.66 | 14:43 |
| *** alexchadin has quit IRC | 14:43 | |
| clarkb | which is worse than just understanding the one | 14:44 |
| mordred | clarkb: I would be *strictly* interested in it as code browsing same as cgit is today | 14:44 |
| prometheanfire | well, comparing commits works | 14:44 |
| prometheanfire | but not tags? | 14:44 |
| clarkb | slaweq: I dont think so because neutron without dvr has different routing needs for the FIP range than neutron with dvr | 14:44 |
| *** aidin has quit IRC | 14:44 | |
| clarkb | slaweq: pretry sure you need to add the multinode networking overlay for br-ex | 14:44 |
| mordred | slaweq: do you want me to put in the autohold on the correct job/project? | 14:44 |
| *** jtomasek has joined #openstack-infra | 14:45 | |
| mordred | or is the stuff from clarkb good for now? | 14:45 |
| slaweq | mordred: I will try what clarkb is suggesting | 14:45 |
| *** bobh has quit IRC | 14:45 | |
| slaweq | will ping You later if I will need Your help again | 14:45 |
| slaweq | thx a lot | 14:45 |
| prometheanfire | smcginnis: looks like it's an open issue (compare tags/branches in gogs) | 14:45 |
| mordred | slaweq: cool | 14:46 |
| prometheanfire | https://github.com/gogs/gogs/issues/3621 | 14:46 |
| clarkb | slaweq: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt#n81 ascii art explaining the neutron scenarios with and without dvr | 14:47 |
| smcginnis | prometheanfire: Seems pretty basic (and easy to implement) | 14:47 |
| prometheanfire | ya | 14:48 |
| clarkb | slaweq: actually that first diagram may be wrong, there is no br-ex vxlan tunnel in the non dvr case I think | 14:48 |
| *** bobh has joined #openstack-infra | 14:48 | |
| prometheanfire | looks like gitea has the same problem | 14:48 |
| slaweq | clarkb: thx, I will check that | 14:49 |
| clarkb | slaweq: but in the dvr case we have to do the br-ex vxlan tunnel so that we can route for the FIP networking on top of regularl cloud netowrking | 14:49 |
| clarkb | slaweq: multi-node-bridge is the role name in openstack-infra/zuul-jobs. By default it creates a bridge with interfaces named br-infra on all the nodes | 14:52 |
| clarkb | slaweq: I think you can bridge br-ex onto that and it will work? | 14:52 |
| clarkb | or you can do like devstack-gate did and just make br-ex the bridge interface name on all the nodes (multi-node-bridge supports setting these variables) | 14:52 |
| *** bobh has quit IRC | 14:53 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on update affiliation endpoint https://review.openstack.org/603824 | 14:54 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on update affiliation endpoint https://review.openstack.org/603824 | 14:55 |
| *** alexchadin has joined #openstack-infra | 14:57 | |
| *** bobh has joined #openstack-infra | 14:59 | |
| clarkb | slaweq: let me know if you have questions on how to piece that together. The best place to start may be looking back to the old legacy dvr job defs | 14:59 |
| clarkb | but I'm happy to help as I can too | 14:59 |
| slaweq | clarkb: ok, thx a lot. I will get back to You if I will need something | 14:59 |
| *** janki has joined #openstack-infra | 15:03 | |
| *** bobh has quit IRC | 15:03 | |
| slaweq | clarkb: so when I connected br-ex with br-infra with patch ports, connectivity works fine :) | 15:07 |
| *** ykarel|away is now known as ykarel | 15:07 | |
| clarkb | slaweq: cool that is what I expected | 15:07 |
| clarkb | slaweq: good to know that piece is working at least :) | 15:07 |
| slaweq | do You think I can set br-infra as external bridge instead of br-ex in job definition then? | 15:07 |
| clarkb | slaweq: you could try it, it is ovs in part because it made integrating it into neutron that way simpler | 15:08 |
| *** janki has quit IRC | 15:08 | |
| *** bobh has joined #openstack-infra | 15:10 | |
| *** Emine has quit IRC | 15:10 | |
| *** bobh has quit IRC | 15:14 | |
| slaweq | clarkb: can You check if something like in https://review.openstack.org/#/c/578796/19/.zuul.yaml in L178 would be enough to replace br-infra with br-ex directly? because then it should also works IIUC, right? | 15:16 |
| *** udesale has joined #openstack-infra | 15:17 | |
| clarkb | slaweq: yes bridge_name to br-ex in the vars is what I would try first | 15:17 |
| slaweq | thx clarkb | 15:17 |
| slaweq | so lets now check for results :) | 15:17 |
| *** alexchadin has quit IRC | 15:19 | |
| slaweq | mordred: You can remove those nodes with neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 | 15:21 |
| slaweq | I think I found what's wrong there, with big help from clarkb :) | 15:21 |
| slaweq | thx to both of You guys :) | 15:21 |
| clarkb | slaweq: maybe you want to take that ascii "art" I made in devstack-gate and give it a new home closer to where the new jobs will live | 15:22 |
| clarkb | slaweq: so that the next person doesn't need to work so hard to find that info :) | 15:22 |
| *** bobh has joined #openstack-infra | 15:22 | |
| slaweq | clarkb: You mean to store it somewhere in neutron repo? | 15:24 |
| *** armax has joined #openstack-infra | 15:25 | |
| *** bobh has quit IRC | 15:27 | |
| clarkb | slaweq: if it doesn't look too ugly as a comment next to the dvr jobs maybe do that? | 15:27 |
| *** dtantsur is now known as dtantsur|brb | 15:28 | |
| slaweq | clarkb: that is IMO good idea, I will add comment with link to this description there | 15:28 |
| slaweq | but let's wait for result now :) | 15:28 |
| slaweq | thx once again for help | 15:28 |
| fungi | smcginnis: one reason we switched to cgit is that it's also what kernel.org uses. it's possible newer cgit would also be nicer | 15:28 |
| smcginnis | fungi: Not that I saw. :) | 15:29 |
| *** jamesmcarthur has quit IRC | 15:30 | |
| *** eernst has joined #openstack-infra | 15:30 | |
| fungi | but yeah, i'm open to alternatives. we can even host them in parallel for a while fairly easily | 15:30 |
| *** hasharAway is now known as hashar | 15:30 | |
| mordred | fungi: ++ | 15:30 |
| *** bobh has joined #openstack-infra | 15:31 | |
| fungi | though i would eventually like to add some fancy rewrite rules to allow us to use the same urls for browsing and git remotes | 15:31 |
| mordred | fungi: I would also like to do that | 15:31 |
| fungi | i have a set i use for some of my personal projects and haven't run into any issues yet | 15:31 |
| *** jamesmcarthur has joined #openstack-infra | 15:32 | |
| *** chkumar|ruck is now known as chkumar|off | 15:34 | |
| *** bobh has quit IRC | 15:36 | |
| *** gyee has joined #openstack-infra | 15:37 | |
| *** bobh has joined #openstack-infra | 15:37 | |
| *** bobh has quit IRC | 15:42 | |
| *** panda has quit IRC | 15:42 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add support for authentication/STARTTLS to SMTP https://review.openstack.org/603833 | 15:42 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add support for authentication/STARTTLS to SMTP https://review.openstack.org/603833 | 15:44 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add support for authentication/STARTTLS to SMTP https://review.openstack.org/603833 | 15:46 |
| *** dtantsur|brb is now known as dtantsur | 15:46 | |
| *** pcaruana has joined #openstack-infra | 15:47 | |
| *** bobh has joined #openstack-infra | 15:47 | |
| *** bobh has quit IRC | 15:47 | |
| *** bobh has joined #openstack-infra | 15:47 | |
| *** bdodd has joined #openstack-infra | 15:50 | |
| *** yamamoto has quit IRC | 15:53 | |
| *** ykarel is now known as ykarel|away | 15:56 | |
| clarkb | fungi: are you in a spot to make the git review release today? | 15:57 |
| clarkb | apparently gerrit is waiting on us before making their next release | 15:57 |
| fungi | clarkb: yeah, was there anything urgent to get in, or should we just plan to have a couple of releases closer together? | 15:58 |
| clarkb | fungi: I think the most important item is whatever fix we need for gerrit (is that refs/for vs refs/otherthing) ? | 15:58 |
| fungi | i thought it already merged... double checking | 16:00 |
| clarkb | ya I think it did, just double checking there aren't other fixes that gerrit needs | 16:00 |
| *** diablo_rojo has joined #openstack-infra | 16:00 | |
| clarkb | according to the email thread that is the one | 16:00 |
| clarkb | I think we can do multiple releases if we need to | 16:01 |
| clarkb | get this fix out for gerrit to release | 16:01 |
| *** ykarel|away has quit IRC | 16:01 | |
| fungi | ssbarnea and others have pointed out 195043 601251 200860 480267 as further possibilities for inclusion | 16:02 |
| clarkb | I'll take a quick look | 16:02 |
| clarkb | https://review.openstack.org/#/c/601251/ seems appropriate given its similarity to the refs/for issue | 16:03 |
| fungi | agreed. i haven't had time to dig into the implementation but if it lgty it already has a +2 from jhesketh | 16:03 |
| clarkb | https://review.openstack.org/#/c/480267/3 is potentially backward incompatible change for scripts | 16:04 |
| clarkb | I don't think we should have ^ in this release as a result | 16:04 |
| clarkb | https://review.openstack.org/#/c/200860/ is -1'd so maybe we can skip it too | 16:04 |
| clarkb | https://review.openstack.org/#/c/195043/ is a new feature so that can probably go in with 480267 if we want it | 16:04 |
| clarkb | tldr I'll review 601251 now | 16:05 |
| fungi | thanks for the quick overview | 16:05 |
| fungi | still trying to catch up this morning unfortunately | 16:05 |
| clarkb | I'm going to test that change | 16:06 |
| clarkb | it adds a comma to the push command I'm not sure is correct | 16:07 |
| *** e0ne has joined #openstack-infra | 16:07 | |
| clarkb | at least the previous +='s didn't add commas | 16:07 |
| fungi | i need to go drop christine off at work real fast and then can push the tag one way or the other | 16:11 |
| * fungi will brb | 16:11 | |
| *** agopi has quit IRC | 16:12 | |
| *** e0ne has quit IRC | 16:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/git-review master: Do Not Merge testing https://review.openstack.org/603842 | 16:19 |
| clarkb | fungi: I left comment on https://review.openstack.org/#/c/601251/3 but change lgtm. If my comments makes sense to you I would say approve it then we can tag that for a new release | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** olivierb has quit IRC | 16:26 | |
| *** yamamoto has joined #openstack-infra | 16:26 | |
| *** pcaruana has quit IRC | 16:27 | |
| *** udesale has quit IRC | 16:28 | |
| *** e0ne has joined #openstack-infra | 16:31 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 16:32 |
| *** zhangfei has joined #openstack-infra | 16:32 | |
| clarkb | I'm going to grab breakfast now | 16:32 |
| *** jpich has quit IRC | 16:32 | |
| fungi | thanks, back and taking a look now | 16:34 |
| *** e0ne has quit IRC | 16:35 | |
| *** ginopc has quit IRC | 16:35 | |
| *** holser__ has quit IRC | 16:36 | |
| *** ykarel has joined #openstack-infra | 16:36 | |
| *** panda has joined #openstack-infra | 16:38 | |
| fungi | clarkb: what do you think about 601823? seems like a fairly trivial bug fix we might also want to stuff in here | 16:41 |
| *** ramishra has quit IRC | 16:42 | |
| *** bobh has quit IRC | 16:43 | |
| fungi | looks like it may be redundant with your 532359 change now that i look closer | 16:45 |
| *** agopi has joined #openstack-infra | 16:50 | |
| clarkb | fungi: the int types thing was important I remember | 16:51 |
| clarkb | because gerrit was giving it as a json string but now a json int | 16:51 |
| *** derekh has quit IRC | 16:51 | |
| clarkb | and you get type mismatches without that depending on the gerrit version | 16:51 |
| clarkb | for that reason I'm inclined to go with my change (but I am clearly a biased opinion :) ) | 16:51 |
| clarkb | fungi: I'd rather not approve my own change there though. Maybe you want to treat electrofelix +1 as good enough? | 16:53 |
| fungi | do you think we should include it in today's release, or defer and make sure it solves the broken -m behavior? | 16:53 |
| clarkb | I think I tested it when I wrote it and it worked | 16:54 |
| clarkb | its definitely broken without that chagne so if we get it in now under the assumption it fixes it the worst that can happen is it will still be broken | 16:54 |
| *** zhangfei has quit IRC | 16:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-python36 job to Neutron Grafana dashboard https://review.openstack.org/595573 | 16:56 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul master: Assure that status tooltip is displayed on entire row https://review.openstack.org/603504 | 16:59 |
| otherwiseguy | is it just me or is the gate, like, super slow. Like things in for > 12 hours slow? | 16:59 |
| clarkb | jroll: about? curious if you had more info on the edge glance needs. Thinking that maybe nodepool actually solves that problem for them (maybe not in the most efficient manner) | 17:00 |
| fungi | otherwiseguy: one of our largest donors is performing upgrade maintenance and we've had to disable them temporarily | 17:00 |
| clarkb | otherwiseguy: top of the integrated openstack gate has been around for almost a day. its ^ as well as flaky testing | 17:00 |
| otherwiseguy | ah. that would do it. :D | 17:00 |
| fungi | they thought they were going to be finished sooner, but it's taking longer than they expected | 17:00 |
| logan- | heh node requests at a 90 day high http://grafana.openstack.org/d/T6vSHcSik/zuul-status?panelId=17&fullscreen&orgId=1&from=now-90d&to=now | 17:01 |
| fungi | otherwiseguy: yeah, for gate resets in particular, you can probably blame drunken coding at the ptg ;) | 17:01 |
| clarkb | imacdonn: I meant to clarify further last night but it was late for me. That is how zuul operates. It builds a speculative future state based on code reviewers approving code (it effectively serializes all of those changes in the order zuul hears about the approvals), assumes they will all work because hey the humans said they will and tests them that way. If they all pass testing we can merge all of | 17:01 |
| clarkb | them together | 17:01 |
| otherwiseguy | so...much...ptg...beer | 17:01 |
| clarkb | imacdonn: the trouble is when they start failing we have to discard those results, build a new speculative state that removes the broken change then start over | 17:02 |
| clarkb | imacdonn: the way to improve this is to write more reliable tests/software and avoid the gate restarts | 17:02 |
| fungi | otherwiseguy: one of the risks of holding a working event two blocks from a brewery i guess | 17:02 |
| EmilienM | hello infra, can I get +A on puppet-placement addition in project-config, thanks https://review.openstack.org/#/c/602869/ | 17:02 |
| clarkb | we tend to see it happen in waves. Gate will be incredibly reliable then degrade people make changes and not fix tests/software | 17:02 |
| clarkb | eventually it gets broken enough that people get annoyed and then fix it | 17:03 |
| clarkb | unfortunately without the likes of mtreinish sdague jogo and mriedem curating things they tend to be unhappy more often than in the past | 17:03 |
| otherwiseguy | clarkb, but it is so easy to just type 'recheck'! :D | 17:04 |
| clarkb | all that to say if this bugs you, the best way to help is to start identifying bugs, http://status.openstack.org/elastic-recheck/data/integrated_gate.html, then fixing them http://status.openstack.org/elastic-recheck/gate.html | 17:04 |
| otherwiseguy | i mean, it'll probably eventually work, right? :p | 17:04 |
| jroll | clarkb: short version is, they want to run more localized glance servers to reduce bandwidth usage... there's a few different options. good point on nodepool, that might actually be reasonable. definitely more reasonable than building python apps that sync data between mysql instances. :) | 17:05 |
| clarkb | otherwiseguy: we'ev actually found cases where the bug was clearly in the change that was rechecked 50 times before it merged and it only passed the 1/50% of the time | 17:05 |
| clarkb | otherwiseguy: :/ | 17:05 |
| otherwiseguy | clarkb, I don't doubt that. | 17:05 |
| clarkb | jroll: ya its not perfect and you have to accept some skew, but it will aggressively do its best to make sure every cloud it knows about has the images it knows about :) | 17:05 |
| jroll | clarkb: yeah, anything we do here is going to be eventually consistent. thanks, I'll add that to... something | 17:06 |
| *** anteaya has joined #openstack-infra | 17:06 | |
| clarkb | jroll: we've also got the ansible cloud launcher stuff which will upload images without thinking about how they were built | 17:07 |
| clarkb | it is far less aggressive thoug | 17:07 |
| jroll | yep | 17:07 |
| clarkb | logan-: everyone is excited to get to work after the PTG I guess | 17:07 |
| *** trown is now known as trown|lunch | 17:09 | |
| openstackgerrit | Merged openstack-infra/git-review master: Use new %topic=XXXX syntax for topic pushes https://review.openstack.org/601251 | 17:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add support for async tasks https://review.openstack.org/603850 | 17:11 |
| clarkb | there have been ~16 unittest failurse in the integrated openstack gate over the last ~day | 17:13 |
| clarkb | mriedem: looks like the majority of them may be nova changes? is that known? | 17:14 |
| *** zhangfei has joined #openstack-infra | 17:18 | |
| prometheanfire | gate just busy today? | 17:20 |
| imacdonn | ^^ :) | 17:20 |
| imacdonn | clarkb: attempting to digest above ... I guess I kinda/sorta get it, but would probably need to understand more about the internals of zuul than I have capacity for today ;) | 17:21 |
| mordred | imacdonn: digesting zuul internals one chunk at a time is recommended ;) | 17:22 |
| imacdonn | :) | 17:23 |
| clarkb | imacdonn: the short version of it is if there is a test failure we have to restart everything without the failing change in consideration for merging | 17:23 |
| clarkb | imacdonn: this process is optimized for systems that have working test suites and right now we don't appear to have that | 17:24 |
| imacdonn | clarkb: does that apply across projects .. e.g. if a nova job fails, does glance have to start over ? | 17:24 |
| clarkb | imacdonn: it applies to all projects in the same gate queue. In this case nova cinder glance keystone swift neutron (and probably a couple others) share a queue | 17:24 |
| imacdonn | clarkb: OK, that explains some of what I'm seeing (affecting both cinder and glance) | 17:26 |
| clarkb | imacdonn: all of the tripleo projects are in a separate shared queue, and so on | 17:26 |
| imacdonn | right | 17:26 |
| *** sambetts is now known as sambetts|afk | 17:27 | |
| prometheanfire | and project-config :P | 17:27 |
| *** jpena is now known as jpena|off | 17:30 | |
| clarkb | looks like glance unittests just timed out on inap. I don't see anything pointing at the test node. I looks like the glance tests just stop | 17:33 |
| clarkb | I wonder if we are going to have to accept this type of flakyness as we ramp up the python3 efforts | 17:33 |
| clarkb | dhellmann: ^ we might try to track this as a metric (even if its just success rate of python3X tests vs python27 tests?) | 17:33 |
| *** jamesmcarthur has quit IRC | 17:34 | |
| dhellmann | clarkb : when I find some time I was going to try to pull some stats about the number of rechecks needed to land the patches for the migration | 17:34 |
| openstackgerrit | Merged openstack-infra/git-review master: Fix compare_review's use of fetch_review https://review.openstack.org/532359 | 17:34 |
| dhellmann | if teams find the new 3.6 tests to be more flakey than the existing 3.5 tests then that seems like an indication of a potential issue with the language version | 17:36 |
| clarkb | fungi: ^ tagging time? | 17:36 |
| fungi | yep, was just double-checking that all the changes we approved have merged | 17:36 |
| clarkb | dhellmann: this glance failure was python35 and nova seems to have a few python35 unittest failures too. I think in some cases the set of tests we ran under python3 was constrained and now we are constraining less ( I could be wrong about that though ) | 17:36 |
| dhellmann | that did used to be true for many teams. it may have changed recently. | 17:37 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Clean up vestigal scripting in cmd.py https://review.openstack.org/567297 | 17:39 |
| clarkb | reminds me a lot of gate behavior around feature freeze fwiw | 17:39 |
| clarkb | oh this job that timed out against glance is a functional python35 job not unittests | 17:40 |
| clarkb | which is likely new | 17:40 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Allow choosing which field to use as author when naming branch https://review.openstack.org/444574 | 17:42 |
| fungi | clarkb: ssbarnea: okay, i think i'm ready to tag git-review 185fb8d (current master branch tip) as 1.27.0... any last-minute objections? | 17:43 |
| *** aidin has joined #openstack-infra | 17:45 | |
| *** zhangfei has quit IRC | 17:45 | |
| clarkb | fungi: not from me | 17:48 |
| *** aidin has quit IRC | 17:49 | |
| ssbarnea | clarkb: there are other changes pending but this should not stop you from doing it, go! we have no reason not to make another release in one week. | 17:49 |
| *** Tim_ok has joined #openstack-infra | 17:50 | |
| fungi | ssbarnea: agreed, i mostly want to stop holding up the gerrit community from making a new release. it's very generous that they wanted to block on modernization in git-review | 17:50 |
| ssbarnea | i am always running git-review from master branch, so I doubt we would break the world. | 17:51 |
| mriedem | clarkb: like this? http://logs.openstack.org/72/600372/1/gate/openstack-tox-py35/dcfd363/testr_results.html.gz | 17:51 |
| clarkb | mriedem: ya that was one of the changes I found via e-r uncategorized | 17:52 |
| mriedem | i've seen those tests fail in weird ways before, but not in a rash | 17:52 |
| clarkb | mriedem: there are a whole back in the last day, day and a half that seem to be nova changes in elastic-recheck uncategorized list for the gate | 17:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Create the OpenStack discussion mailing list https://review.openstack.org/602781 | 17:53 |
| clarkb | (some are not noav too, but majority appaer to be nova) | 17:53 |
| mriedem | my guess would be eventlet something or other | 17:54 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.ResourceClosedError%3A%20This%20result%20object%20does%20not%20return%20rows.%20It%20has%20been%20closed%20automatically.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d doesn't hit much | 17:55 |
| roman_g | Hello all. With Zuul queue being so long, I'd like to ask the following question: is there a "fast lane" queue in Zuul to run after merge? "Post" queue is quite slow. All I need is to add a tag with commit id to existing docker image (get commit id->pull image->add tag->push new tag), and I would expect it to happen as fast as possible right after merge, so that there is always an image in | 17:56 |
| roman_g | container repository with a tag corresponding to the latest git commit id. | 17:56 |
| clarkb | mriedem: http://logs.openstack.org/03/602403/1/gate/openstack-tox-py35/b4c9214/testr_results.html.gz is another nova failure | 17:56 |
| mriedem | yeah there is a IndexError: tuple index out of range in there | 17:57 |
| mriedem | but i don't know where that is coming from | 17:57 |
| AJaeger | clarkb, fungi, we also have less capacity right now with with OVH BHS down - missing 159 nodes - that explains part of the backlog | 17:58 |
| clarkb | roman_g: the current prioritization is gate,release|tag > check > post,periodic | 17:59 |
| roman_g | "pull image->add tag->push new tag" could also be replaced with just API call, I've just checked docs | 17:59 |
| roman_g | clarkb: thank you. | 18:00 |
| clarkb | roman_g: you can potentially run that as a null nodeset job then, which won't need nodes to be assigned which will allow it to run quickly | 18:00 |
| clarkb | roman_g: the gotcha is you have to operate in a very constrained environment where basically all you can do is talk to external services | 18:00 |
| roman_g | clarkb: oh, that's cool. I didn't think it's possible. | 18:01 |
| clarkb | you cannot install additional software, but python should be there for you to talk http for example | 18:01 |
| clarkb | also you may have to use native ansible? we cannot shell on localhost or can we? | 18:01 |
| clarkb | dpawlik: I assume we shouldn't enable ovh again? | 18:02 |
| roman_g | clarkb: so my usecase fits pretty well then: get latest commit id, and make api call (Ansible works perfectly). | 18:02 |
| clarkb | dpawlik: specifically bhs1? | 18:02 |
| clarkb | roman_g: ya that should work | 18:02 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/603282/ and https://review.openstack.org/603199 (will recheck once 603282 is in), please? | 18:02 |
| roman_g | clarkb: which queue would you recommend then? release? | 18:02 |
| clarkb | mriedem: http://logs.openstack.org/94/603194/1/gate/openstack-tox-py35/e37d161/testr_results.html.gz is another db looking failure like nova's but in cinder | 18:02 |
| fungi | clarkb: ssbarnea: https://pypi.org/project/git-review/ now shows 1.27.0 as current | 18:03 |
| fungi | i'll follow up to the ml thread in a sec | 18:03 |
| roman_g | clarkb: or still "post", but without nodes? | 18:03 |
| clarkb | roman_g: post, the precedence is for node assignments, because there are no node assignments I think you can run immediately | 18:03 |
| clarkb | roman_g: yup exactly that | 18:03 |
| roman_g | clarkb: cool. Thank you! | 18:04 |
| ssbarnea | lets hope we don't have to unrelease it, previous one was like 10 months ago,... | 18:04 |
| clarkb | ssbarnea: chances are we will roll forward | 18:04 |
| clarkb | rather than delete a release | 18:04 |
| clarkb | roman_g: if you get something working it may be worth sharing as part of the zuul-jobs roles/jobs if generally applicable | 18:04 |
| mriedem | clarkb: yeah those all use the same oslo.db opportunistic db test fixtures for walking the schema migrations | 18:05 |
| mriedem | i seem to remember years ago we had to bump the timeout on those tests, | 18:05 |
| mriedem | maybe we need to do that again | 18:05 |
| clarkb | roman_g: other things to note if you don't specify a nodeset you get the default nodeset. So you have to actually specify a nodeset that has no nodes in it. Also you have to explicitly use localhost and not all in the ansible to run on localhost | 18:05 |
| roman_g | clarkb: how can I disable node assigned for a specific job? I think by default all my jobs are run in a nodeset | 18:06 |
| ssbarnea | clarkb: this only if the fix is quick, hiding from pypi should be first action if serious bug is found. had to do this only twice so far with jira library. | 18:06 |
| clarkb | roman_g: let me find an example | 18:06 |
| roman_g | clarkb: -job: ... nodeset: localhost ?? | 18:06 |
| mriedem | clarkb: yeah https://review.openstack.org/#/c/370805/ | 18:08 |
| *** trown|lunch is now known as trown | 18:08 | |
| clarkb | roman_g: https://git.openstack.org/cgit/openstack-infra/system-config/tree/.zuul.yaml#n211 like that | 18:08 |
| clarkb | mriedem: is that due to accumulation of migrations? we don't roll them up anymore I guess? | 18:09 |
| mriedem | might be part of it, | 18:09 |
| mriedem | and just slower nodes in general b/c of spectre/meltdown patces | 18:09 |
| mriedem | *patches | 18:09 |
| clarkb | newer kernels do seem to help with that at least | 18:10 |
| clarkb | (maybe this is the motivation to move to bionic more quickly | 18:10 |
| clarkb | at least with infra's testing on rax we got much better performance with latest xenial HWE kernel compared to older HWE kernels | 18:11 |
| logan- | i wonder how much difference it would make to run hwe kernels on our nodepool hvs | 18:11 |
| roman_g | clarkb: thank you!! | 18:13 |
| fungi | logan-: we're happy to help figure out what the performance difference looks like if we can | 18:14 |
| fungi | mapping builds back to hypervisor hosts and coming up with baseline performance numbers is nontrivial though | 18:14 |
| clarkb | graphite has job timing data per cloud | 18:15 |
| clarkb | we'd probably just look at that for something like nova unittests | 18:15 |
| clarkb | and see if we notice a change | 18:15 |
| fungi | it does, but yeah depends on whether nova unittests have a consistent performance profile | 18:15 |
| clarkb | they should be pretty consistent on specific clouds I would expect | 18:16 |
| fungi | and filter by success only obviously | 18:16 |
| clarkb | devstack/tempest jobs are likely to be better measures of meltdown performance impact though | 18:16 |
| fungi | on the guest side or on the host side? | 18:16 |
| fungi | or both? | 18:17 |
| clarkb | guest side for sure since there are plenty of syscalls on the guest kernel | 18:17 |
| clarkb | I'm not sure how that maps into the ways kvm was affected | 18:17 |
| mriedem | clarkb: well https://bugs.launchpad.net/cinder/+bug/1793364 | 18:17 |
| openstack | Launchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate" [High,Confirmed] | 18:17 |
| logan- | btw clarkb, I think you were right that the limestone mirror issue was due to a full disk. It seems like the ELK stuff we deployed last week was eating too much local storage. we are moving it to rbd volumes. i also noticed several nova failures you just posted were on limestone nodes, and in general I see a spike in failures from the 15th thru the 18th on limestone | 18:17 |
| logan- | https://i.imgur.com/B5Zwj4a.jpg | 18:17 |
| mriedem | and it's not a simple timeout, | 18:17 |
| logan- | (gist is here: https://gist.github.com/logan2211/76e7a86fccb04a4db9de0ba96fb83f4e) | 18:18 |
| mriedem | here is a failure | 18:18 |
| mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED | 18:18 |
| mriedem | from a passing run: | 18:18 |
| mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok | 18:18 |
| clarkb | logan-: interesting | 18:18 |
| clarkb | mriedem: woah | 18:18 |
| mriedem | so we're clearly losing a context switch or something with eventlet | 18:18 |
| fungi | that's going to be fun to track down :/ | 18:18 |
| clarkb | logan-: I guess that is because our disks are thin provisioned so even though our mirror wasn't using much disk on the hv the hv ran out disk from noisy neighbor ELK stack? | 18:19 |
| logan- | yeah | 18:20 |
| clarkb | makes sense | 18:20 |
| * clarkb upgrades git-review to be new release tester | 18:20 | |
| fungi | bleeding edge! | 18:20 |
| clarkb | I need to step away from the computer for a few minutes, but when I get back I'll draft up a dev list status update on why things are slow (BHS1 maintenance, flaky tests, etc) | 18:21 |
| clarkb | hopefully that will help people understand what is going on | 18:21 |
| fungi | thanks clarkb! | 18:22 |
| fungi | i'm about to context-switch to configuring the future of our mailing lists following the next ansible pulse | 18:22 |
| fungi | and then i should be able to send out the announcement about that plan | 18:22 |
| yumiriam | smcginnis: hi, i submitted this patch: https://review.openstack.org/#/c/599720, and i investigated why lvm lio job was failing | 18:23 |
| ssbarnea | fungi: Thanks for git-review work! maybe it would be a good idea to add two more cores to git-review, so you would not become overloaded (maybe even documenting them somewhere so people will know who to add as reviewer, i am sure that not everyone knows how to dig gerrit ACL in order to find people that can review). | 18:25 |
| *** ykarel has quit IRC | 18:26 | |
| yumiriam | smcginnis: i think i figured out what was causing the problem, i'll have to change the tempest_roles in tempest configuration, could you help me to do it? | 18:26 |
| *** anteaya has quit IRC | 18:27 | |
| *** zhangfei has joined #openstack-infra | 18:32 | |
| fungi | ssbarnea: yes, i agree. we should have a talk with the infra ptl about it when he's not quite so busy ;) | 18:33 |
| mriedem | clarkb: welp, i don't have any good ideas on how to fingerprint this in e-r | 18:33 |
| mriedem | the indexerror is probably the killer | 18:35 |
| mriedem | but i don't know where it starts | 18:35 |
| mriedem | and we can't do multi-line fingerprinting for context | 18:35 |
| clarkb | mriedem: if it is a traceback the whole traceback should be indexed as one event ? | 18:36 |
| mriedem | only in screen logs | 18:36 |
| mriedem | not console | 18:36 |
| *** ijw has joined #openstack-infra | 18:36 | |
| clarkb | ah | 18:36 |
| mriedem | my guess is the indexerror comes from reading a buffer in eventlet | 18:38 |
| mriedem | that's in the stacktrace | 18:38 |
| clarkb | glance python2 functional job just reset the whole gate so not python3 specific there | 18:39 |
| *** ijw has quit IRC | 18:42 | |
| *** zhangfei has quit IRC | 18:45 | |
| *** vkmc is now known as vkmc|afk | 18:49 | |
| roman_g | Question: where can I see logs for zuul "post" jobs being published on merge? | 18:57 |
| mriedem | clarkb: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 18:57 |
| mriedem | that will surely get our categorization numbers up | 18:57 |
| mriedem | count per 1h | (1890 hits) | 18:57 |
| clarkb | roman_g: easiset way to find them is from the builds tab of the zuul status page | 18:57 |
| clarkb | mriedem: that is a lot of hits | 18:57 |
| clarkb | mriedem: I'm writing email to the dev list about helping with e-r and fixing some of these bugs | 18:57 |
| clarkb | mriedem: I'll put it on etherpad if you are interested in reading it first | 18:57 |
| mriedem | heh to make sure you are PC enough? :P | 18:58 |
| mriedem | WWMD | 18:58 |
| clarkb | mriedem: more so that you don't feel I've thrown you to the wolves :P | 18:58 |
| mriedem | are there wolves? | 18:58 |
| mriedem | i remember sending several "the gate is melting and it's ALL YOUR FAULT" emails over the years | 18:58 |
| mriedem | but like you said, in the old days with mtreinish and jogo | 18:59 |
| *** ijw has joined #openstack-infra | 18:59 | |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| mriedem | by all means though, raise the alarm | 18:59 |
| clarkb | mriedem: "If you'd like to help let mriedem or myself know and we'll gladly work with you to get elasticsearch queries added to elastic-recheck. We are likely less help when it comes to fixing functional tests in Glance, but I'm happy to point people in the right direction for that as much as I can. | 19:01 |
| clarkb | er that was meant to have an end quote | 19:01 |
| clarkb | but thats the tldr of throwing you to the wolves :) | 19:01 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for mysql opportunistic test bug 1793364 https://review.openstack.org/603874 | 19:02 |
| openstack | bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] https://launchpad.net/bugs/1793364 | 19:02 |
| roman_g | clarkb: thanks. Actually I found it easier to find logs by filtering on Builds page. | 19:02 |
| roman_g | *filtering by job name | 19:02 |
| mriedem | clarkb: e-r review wolves, sure | 19:02 |
| mriedem | i'm down | 19:02 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: convert docs to PTI https://review.openstack.org/559396 | 19:04 |
| AJaeger | mriedem: your change is not needed - but there's more to fix ^ | 19:07 |
| AJaeger | mriedem: will you take care of it? | 19:07 |
| mriedem | AJaeger: it's not my change, | 19:09 |
| mriedem | i'm just rebasing it | 19:09 |
| mriedem | AJaeger: if you want it, it's yours | 19:09 |
| AJaeger | ;) | 19:10 |
| mriedem | clarkb: is this a known issue? http://logs.openstack.org/17/595317/1/gate/build-openstack-sphinx-docs/b8849f2/job-output.txt.gz#_2018-09-18_18_33_45_353493 | 19:12 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22rsync%20error%3A%20unexplained%20error%20(code%20255)%20at%5C%22&from=7d | 19:12 |
| clarkb | mriedem: I think that is a limestone node (based on ipv6 usage) and I think logan- said he was fixing some stuff there? | 19:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/elastic-recheck master: convert docs to PTI https://review.openstack.org/559396 | 19:13 |
| clarkb | mriedem: it is odd that it timed out then failed to reconnect | 19:13 |
| clarkb | logan-: ^ | 19:13 |
| AJaeger | mriedem: done ^ | 19:13 |
| mriedem | definitely hitting most on limestone-regionone | 19:14 |
| mriedem | AJaeger: thanks | 19:14 |
| mriedem | clarkb: is there a bug for that? | 19:14 |
| clarkb | mriedem: I don't think so | 19:14 |
| clarkb | mriedem: I think we only just started to get a handle on it a few minutes ago in scrollback (about an hour ago) | 19:15 |
| logan- | ya after I found the ELK disk usage stuff yesterday I moved some things around to resolve it. today looks much smoother: https://i.imgur.com/B5Zwj4a.jpg | 19:15 |
| clarkb | mriedem: logan- we can stick a query in e-r to make sure it has been resolved | 19:15 |
| mriedem | i've got it | 19:16 |
| mriedem | https://bugs.launchpad.net/openstack-gate/+bug/1793370 | 19:16 |
| openstack | Launchpad bug 1793370 in OpenStack-Gate ""Collect sphinx build html" fails with "rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]" on limestone nodes" [Undecided,New] | 19:16 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for ansible ssh rsync fail bug 1793370 https://review.openstack.org/603878 | 19:18 |
| openstack | bug 1793370 in OpenStack-Gate ""Collect sphinx build html" fails with "rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]" on limestone nodes" [Undecided,New] https://launchpad.net/bugs/1793370 | 19:18 |
| AJaeger | dpawlik: how is OVH BHS coming along? You mentioned some time ago 1 or 2 h which is long over - is it more complicated or are you done? | 19:24 |
| *** gema has joined #openstack-infra | 19:32 | |
| clarkb | AJaeger: my bhs1 vm just stopped responding then started again. Reading between the lines I would assume that was a live migration related to updates | 19:32 |
| clarkb | AJaeger: possibly still in progress given that | 19:33 |
| AJaeger | clarkb: dpawlik's ask for 1 or 2 hours was 6 hours ago, so that's why I was asking. Yes, might be they had some surprises ;( | 19:36 |
| * AJaeger signs off for today | 19:37 | |
| fungi | i hear upgrading openstack isn't simple | 19:38 |
| clarkb | fungi: they are doing the fun route of jumping multiple versions too aiui | 19:39 |
| fungi | doesn't help that they seem to be going from juno to newton | 19:39 |
| fungi | yeah, that | 19:39 |
| fungi | well, i think i have the initial openstack-discuss@l.o.o configuration done. i guess it's time to write up the announcement about it | 19:40 |
| fungi | we said we should subscribe the old lists to the new list around october 24 (two weeks before the summit) and then turn off the old lists around november 21 (the week after the summit) right? | 19:44 |
| clarkb | ssbarnea: fungi as far as more core reviewers I'm happy to add them. I think we should be careful to get fixes like https://review.openstack.org/#/c/428700/6 in over adding new features like https://review.openstack.org/#/c/195043/ | 19:44 |
| clarkb | we've made git-review fairly stable and it doesn't break often for existing users (but as in the case of that one bug fix may not work for new users) | 19:44 |
| *** anteaya has joined #openstack-infra | 19:45 | |
| clarkb | and from there maybe make the test suite a bit more robust and easier/quicker to run. THen consider features? | 19:45 |
| fungi | testing it against newer gerrit was also suggested as something worth prioritizing, i think | 19:46 |
| *** roman_g has quit IRC | 19:46 | |
| *** zhangfei has joined #openstack-infra | 19:46 | |
| clarkb | ++ | 19:47 |
| clarkb | basically I'd like to see is get our feet wet again on the project by fixing bugs and building confidence in testing | 19:47 |
| *** roman_g has joined #openstack-infra | 19:47 | |
| clarkb | if we end up in a good spot for that then consider features again. Since I think part of why we stopped doing feature work was we broke too many people too often in the past | 19:48 |
| fungi | well, that coupled with not having any appreciable testing | 19:51 |
| clarkb | ssbarnea: are you volunteering ? :) | 19:51 |
| clarkb | zxiiro may be interested as well as electrofelix? | 19:51 |
| zxiiro | happy to help if I'm needed. We depend extensively on git-review so makes sense that we spend some cycles helping out there. | 19:53 |
| fungi | my only real reservation about adding more core reviewers is to make sure we share the opinion that adding more features to git-review needs to meet a very high bar these days to avoid scope creep (and that there are even places where we might do well to deprecate/remove some existing features) | 19:56 |
| *** pfallenop has joined #openstack-infra | 19:56 | |
| fungi | keeping focus on it doing a specific set of tasks very well, and blending with use of other common posix/unix command-line utilities via pipes and so on | 19:58 |
| fungi | and considering that some sorts of possible features may be better implemented as separate git subcommands rather than more git-review options | 19:59 |
| *** zhangfei has quit IRC | 19:59 | |
| clarkb | or `use gertty` | 19:59 |
| clarkb | which is excellent software for doing all your gerrit activites on the command line | 19:59 |
| fungi | for fancy use cases, i definitely feel like more robust gerrit clients are a better option | 20:00 |
| fungi | git-review is intended as a streamlined tool for pushing one or a series of commits to a gerrit for review | 20:00 |
| notmyname | clarkb: I just got back from lunch and say your email about the zuul issues. it's fantastic. thanks | 20:00 |
| clarkb | notmyname: I'm glad someone found it useful :) | 20:02 |
| mriedem | clarkb: this should help https://review.openstack.org/#/c/603900/ | 20:06 |
| mriedem | should have been done weeks ago | 20:06 |
| clarkb | mriedem: thanks | 20:06 |
| notmyname | clarkb: with a quick glance at the elastic recheck page, I see one with "swift" in the name. it happened once and two days ago and has since passed. is there a way I can categorize it as "doesn't matter" or transient or resolved or soemthing? | 20:06 |
| mriedem | notmyname: is the bug marked as invalid? | 20:07 |
| mriedem | or which one on http://status.openstack.org/elastic-recheck/index.html ? | 20:07 |
| mriedem | if http://status.openstack.org/elastic-recheck/data/integrated_gate.html#cross-swift-py35 | 20:08 |
| mriedem | it doesn't matter, ignore | 20:08 |
| notmyname | ya, that second one | 20:08 |
| mriedem | http://status.openstack.org/elastic-recheck/index.html is really the "oh shit stuff is on fire" | 20:08 |
| notmyname | ah, ok | 20:08 |
| mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html is what we haven't fingerprinted yet | 20:08 |
| mriedem | Overall Categorization Rate: 15.2% is terrible, | 20:08 |
| mriedem | that should be closer to 50% | 20:08 |
| clarkb | ya the uncategorized page is more of a "hey we should look at things if that categorization rate is hihg" | 20:08 |
| mriedem | meaning, the gate is failing for reasons we aren't tracking | 20:08 |
| clarkb | and start at the top since those have the most occurences | 20:09 |
| mriedem | right | 20:09 |
| clarkb | notmyname: thank you for checking though :) | 20:09 |
| mriedem | the categorization rate should bump up within the hour | 20:09 |
| mriedem | and then we can do another pass | 20:09 |
| notmyname | ah, ok. as a quick heuristic, I'd just searched for "swift" since that's something I can be sortof useful with. the one instance is some issue where zuul couldn't install python (or pyyaml). and it's passed since then. so... ignore it | 20:10 |
| *** jamesmcarthur has quit IRC | 20:10 | |
| clarkb | notmyname: ya if its a one off you can ignore it. e-r is largely oriented around identifying and tracking persistent issues | 20:10 |
| clarkb | but it may also tell us we have no persistent issues and they are all one offs | 20:10 |
| clarkb | and then we have more work to do probably :) | 20:10 |
| fungi | well, we have more work to do regardless... just a question of what we'll be working on ;) | 20:12 |
| clarkb | one thing we might want to add to the uncategorized page is the project that was tested when it failed | 20:13 |
| notmyname | oh wait! | 20:13 |
| clarkb | since we moved to the generic job names with zuulv3 | 20:13 |
| notmyname | clarkb: mriedem: the one thing I saw is in fact the same (or similar to a tracked issue on the e-r page. how do I associate it? | 20:13 |
| clarkb | notmyname: you can update the query in elastic-recheck/queries/$bugnumber.yaml to include matches for your issue | 20:14 |
| notmyname | "cross-swift-py35 : 1 Uncategorized Fails. 0.0% Classification Rate (1 Total Fails)" --->> https://bugs.launchpad.net/openstack-gate/+bug/1449136 | 20:14 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "Pip fails to find distribution for package" [Undecided,New] | 20:14 |
| clarkb | notmyname: to test that go to logstash.openstack.org and plug in the existing query to get the existing result. THen modify as necessary to include the other results | 20:14 |
| mriedem | yeah http://status.openstack.org/elastic-recheck/#1449136 has a logstash link to take you to the existing query directly | 20:14 |
| mriedem | and then you can add an OR to the query | 20:15 |
| mriedem | to see if you get more hits | 20:15 |
| clarkb | this bit is the most hand wavy of the lot (you kind of have to figure out lucene's query syntax) but if you point me at a log line that should match and the existing bug I can try to help come up with some thing | 20:15 |
| notmyname | clarkb: http://logs.openstack.org/83/602183/1/gate/cross-swift-py35/aa80654/job-output.txt.gz#_2018-09-18_08_35_12_688141 | 20:15 |
| clarkb | notmyname: ah yup (thats a known issue we think we fixed in limestone, but lets do this for tracking and learning purposes) | 20:16 |
| *** pfallenop has quit IRC | 20:16 | |
| prometheanfire | notmyname: sup | 20:16 |
| *** pfallenop has joined #openstack-infra | 20:16 | |
| prometheanfire | recheck'd blaming clarkb :D | 20:16 |
| clarkb | notmyname: http://status.openstack.org/elastic-recheck/gate.html#1449136 is the existing bug entry on e-r. if you click the little logstash link under that it opens up the logstash ui with that query | 20:17 |
| notmyname | yeah. I'm there | 20:18 |
| notmyname | trying to figure out why it doesn't already match | 20:18 |
| clarkb | notmyname: I think because in your example there is the serialized newling so its \nNo matching | 20:18 |
| clarkb | and lucene treats that as a different token than No | 20:19 |
| clarkb | So we can update the query to be (message:"No matching distribution found for" OR message:"\nNo matching distribution found for") rest of query here | 20:19 |
| notmyname | yep. running it now | 20:19 |
| clarkb | yup that seems to work. So last step is to update the query in https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries/1449136.yaml to include that new bit | 20:20 |
| clarkb | notmyname: (message:"No matching distribution found for" OR message:"\nNo matching distribution found for") AND tags:console AND voting:1 AND build_queue:gate <- is my query fwiw | 20:21 |
| clarkb | the voting:1 and build_queue:gate stuff is added by elastic-rechech which is why you won't see it in the query file I linked above | 20:21 |
| notmyname | oh, I ddin't have the buil.... ah yes | 20:21 |
| clarkb | (you don't have to add that thwn you edit the file in the elastic-recheck repo) | 20:21 |
| notmyname | yeah, the added message went from 92 matches to ... a lot more than that | 20:22 |
| clarkb | indeed | 20:22 |
| clarkb | (there were issues with the mirror in limestone crashing, we think since fixed. But updating the query will give us a good level of detail into whether or not that is the case | 20:22 |
| fungi | dhellmann: we said we should subscribe the old lists to the new -discuss list around october 24 (two weeks before the summit) and then turn off the old lists around november 21 (the week after the summit) right? or were we looking for tighter timing on that? | 20:23 |
| dhellmann | fungi : I don't remember the dates. Did we have notes on that in the etherpad? | 20:24 |
| clarkb | yes I tried to write those notes down on the etherpad | 20:25 |
| fungi | mm, i guess that would have been the infra pad. checking | 20:25 |
| dhellmann | I think the specific dates you give make sense and mesh with the rougher descriptions I'm seeing in the etherpad | 20:26 |
| dhellmann | a tighter period would work for me, too, but I know there was a lot of concern about giving folks time for the transition | 20:26 |
| fungi | yeah, the etherpad just says "cut over First week of December" | 20:27 |
| dhellmann | it might be confusing in a different way to have too long of a period for the transition | 20:27 |
| *** armax has quit IRC | 20:27 | |
| clarkb | I think the concern with week after summit is US thanksgiving | 20:27 |
| openstackgerrit | John Dickinson proposed openstack-infra/elastic-recheck master: update query 1449136 to match some more queries https://review.openstack.org/603905 | 20:27 |
| notmyname | clarkb: ^^ | 20:27 |
| clarkb | (I personally don't have the holiday concern that others had) | 20:28 |
| zxiiro | clarkb: does gertty support the latest version of Gerrit yet? last time I tried it didn't work for the Gerrit versions LF deploys. | 20:28 |
| fungi | so maybe we resolved to subscribe the new list to the old lists around the summit and then disable them during the first week of december? | 20:28 |
| clarkb | zxiiro: unsure, I don't actually use gertty | 20:28 |
| clarkb | zxiiro: I imagine corvus would like to fix those problems though | 20:28 |
| fungi | zxiiro: when did you last try? | 20:28 |
| dhellmann | fungi : yeah, that seems to match the etherpad better | 20:28 |
| zxiiro | ah ok, yeah it sounds like a cool tool but it didn't work on our Gerrit systems last time I tried. | 20:29 |
| zxiiro | fungi: I tried at the Vancouver Summit. | 20:29 |
| zxiiro | so a few months ago now... | 20:29 |
| clarkb | notmyname: looks great, I will let mriedem double check things as I don't review these as often as he does | 20:29 |
| clarkb | mriedem: can you review https://review.openstack.org/603905 | 20:29 |
| notmyname | here's hoping it helps find some major blockers :-) | 20:29 |
| fungi | zxiiro: yeah, then it's possibly still an issue worth investigating | 20:29 |
| *** jtomasek has quit IRC | 20:29 | |
| *** jamesmcarthur has joined #openstack-infra | 20:30 | |
| mriedem | looking | 20:30 |
| fungi | dhellmann: clarkb: okay, so more like subscribe the new list to the old lists around november 19 (monday after summit) and disable old lists december 7 (first friday of the month)? | 20:31 |
| clarkb | fungi: that wfm | 20:31 |
| fungi | or we could disable on december 2 (first monday) | 20:31 |
| clarkb | actually I think I prefer switching on monday | 20:32 |
| fungi | er, i guess that's december 3 | 20:32 |
| clarkb | people more likely to take it as a todo to address that | 20:32 |
| *** kgiusti has left #openstack-infra | 20:32 | |
| clarkb | whereas on friday you leave it until next week and forget and blah | 20:32 |
| fungi | okay, so phase 1 now to nov 19, phase 2 nov 19 to dec 3, phase 3 begins dec 3 | 20:33 |
| dhellmann | wfm | 20:33 |
| fungi | that's 2 weeks in phase 2 | 20:33 |
| mriedem | notmyname: clarkb: is it just me, or does that bring the hits for that query up from ~62 to ~3741 in 7 days? | 20:33 |
| dhellmann | with an email announcing all of this ~soon | 20:33 |
| clarkb | mriedem: ya there was a limestone sadness, it may match multiples per job though | 20:34 |
| fungi | dhellmann: yes, ~soon is this afternoon or tomorrow morning (i already have the new ml configured) | 20:34 |
| clarkb | mriedem: our mirror was off in limestone for several hours monday and several hours tuesday :( | 20:34 |
| dhellmann | perfect | 20:34 |
| fungi | just making sure i know what dates to communicate | 20:34 |
| clarkb | mriedem: we think we've fixed that, but we can add the query to double check | 20:34 |
| mriedem | ok, +W | 20:34 |
| *** hemna_ has quit IRC | 20:36 | |
| *** priteau has quit IRC | 20:39 | |
| *** anteaya has quit IRC | 20:43 | |
| clarkb | fyi I have approved https://review.openstack.org/#/c/603766/1 to enable bhs1 again | 20:44 |
| clarkb | thank you amorin | 20:44 |
| *** ansmith has quit IRC | 20:45 | |
| *** dklyle has quit IRC | 20:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC"" https://review.openstack.org/603766 | 20:53 |
| clarkb | I think ^ will apply in about half an hour | 20:57 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| fungi | headed to grab an early dinner, but should be back soonish | 21:16 |
| *** hashar has quit IRC | 21:23 | |
| tbarron | hmm, I do '/msg chanserv topic #openstack-manila blah blah' and get 'You are not authorized to perform this operation'. I think this was working, or am I forgetting something obvious? | 21:23 |
| clarkb | tbarron: our channels are +t so only channels ops can set the topic | 21:24 |
| *** jamesmcarthur has quit IRC | 21:24 | |
| clarkb | tbarron: you are listed as having ops in the channel according to chanserv so you should be able to op up, set the topic, then deop | 21:25 |
| *** armax has joined #openstack-infra | 21:25 | |
| clarkb | tbarron: /msg chanserv op #openstack-manila tbarron | 21:25 |
| clarkb | then /msg chanserv deop #openstack-manila tbarron when done iirc | 21:26 |
| *** jamesmcarthur has joined #openstack-infra | 21:26 | |
| *** agopi has quit IRC | 21:28 | |
| *** jamesmcarthur has quit IRC | 21:30 | |
| *** armax has quit IRC | 21:30 | |
| tbarron | clarkb: thanks but it tells me I'm not authorized when I attempt to op up | 21:31 |
| clarkb | tbarron: are you identified with nickserv? | 21:32 |
| tbarron | clarkb: yes | 21:32 |
| tbarron | clarkb: will double check, but I logged in as usual with pass etc. | 21:32 |
| clarkb | nickserv says you are not logged in | 21:33 |
| clarkb | /msg nickserv acc tbarron | 21:33 |
| clarkb | the 1 means not logged in | 21:33 |
| clarkb | we have reenabled bhs1 now. I am going to watch it for a bit | 21:35 |
| *** bdodd has quit IRC | 21:35 | |
| *** armax has joined #openstack-infra | 21:36 | |
| tbarron | clarkb: hmm, i'll shutoff my client and reconnect, when I connect I have to do /quote PASS tom_barron:mypw everytime | 21:37 |
| tbarron | clarkb: and maybe i need to bounce my bouncer (znc) but its configuration worked the last time I changed the topic, during PTG | 21:40 |
| *** sthussey has quit IRC | 21:40 | |
| tbarron | clarkb: not an emergency clearly :) | 21:40 |
| clarkb | tbarron: you should just need to reidentify with nickserv and not bounce your bouncer | 21:40 |
| clarkb | chances are you were disconnected from freenode for some short period then things didn't renegotiate properly on reconnect (could be due to a netsplit or similar) | 21:41 |
| clarkb | tbarron: /msg nickserv identify $pw | 21:41 |
| clarkb | then you can op up and set the topic | 21:41 |
| *** dklyle has joined #openstack-infra | 21:42 | |
| clarkb | BHS1 seems to be happy | 21:45 |
| clarkb | errors about no valid host found now that we are near/at quota but I think that was happening before | 21:45 |
| clarkb | doesn't seem like there were errors prior to that point either which is impressive since we just asked the cloud to boot 150 something instances | 21:46 |
| clarkb | mordred: are you in a place to discuss the zuul CD things I've learned since PTG? | 21:47 |
| clarkb | mordred: I think it would be helpful to run the problems and thoughts on how to address them by someone | 21:47 |
| mordred | clarkb: my brain is pretty toast from today - how about we talk through it as soon as you get up tomorrow and I'll try to not be rabbitholed in weird async loops | 21:48 |
| clarkb | mordred: ok | 21:48 |
| tbarron | clarkb: all works now, thanks | 21:53 |
| *** owalsh has joined #openstack-infra | 21:57 | |
| ianw | tbarron: i use znc too and if you setup sasl the problem disappears | 21:58 |
| ianw | because it authenticates before it starts | 21:58 |
| *** owalsh has quit IRC | 21:58 | |
| tbarron | ianw: and I do have it setup and authenticate at startup, not sure what went wrong | 21:59 |
| tbarron | ianw: sasl | 21:59 |
| tbarron | ianw: never had to separately '/msg nickserv identify $pw' before. | 21:59 |
| tbarron | ianw: some glitch, will see if it repeats | 22:00 |
| ianw | yeah, with many parts doesn't take much to go wrong; similarly i thought i had it working and it stopped too, but i think it was config and docker images and overlays etc too | 22:00 |
| ianw | i am sympathetic to the discussions that irc can be a little difficult to get going reliably :) | 22:01 |
| clarkb | ianw: is this related to running docker comamnds as root? | 22:01 |
| clarkb | fwiw my setup is weechat in screen in the cheapest ovh VM. Seems to have stayed up during their upgrades so far :) | 22:01 |
| tbarron | ianw: it's karma for me not being as sympathetic to that pov as I should have been | 22:01 |
| *** roman_g has quit IRC | 22:02 | |
| ianw | clarkb: not in this case. but i have learnt some lessons about that contributing a few bits to testinfra | 22:02 |
| ianw | the tox there runs a bunch of containers | 22:02 |
| tbarron | i was thinking, oh, just put in a bit of effort one time and forget about it afterwards. And sure enough, I forgot. | 22:02 |
| ianw | and it's not enjoyable when you run "tox" and suddenly your VT switches to an Alpine linux console prompt | 22:02 |
| ianw | how the heck that happens I don't want to know | 22:03 |
| clarkb | ianw: this is one reason we've pushed back on needing root to run tox in openstack land | 22:07 |
| clarkb | turns out you can do all sorts of terrible things this way :( | 22:07 |
| ianw | after reading a bit, it seems that the reason you need to be root for docker in fedora is that using a "docker" group is really just root by another, much less obvious name | 22:08 |
| ianw | https://developer.fedoraproject.org/tools/docker/docker-installation.html | 22:09 |
| clarkb | yup its like being in the kvm/libvirt group | 22:09 |
| clarkb | you can abuse the daemon to gain root | 22:09 |
| clarkb | this is actually one of my concerns with dox vs tox | 22:10 |
| clarkb | dox requires you to give the tests root essentially | 22:10 |
| clarkb | whereas with tox you don't have that and can isolate it under a normal user as long as the tests themselves don't need that access | 22:10 |
| ianw | that's ok, let's just isolate the docker tests in a VM! | 22:10 |
| clarkb | notmyname: do you have a link to the ansible + zuul syntax error you had that OSA table helped you debug? | 22:15 |
| notmyname | clarkb: yes... maybe? I know the patch that caused it | 22:15 |
| clarkb | notmyname: I can find it from there (curious beacuse other users are reporting that syntax errors result in no logs and that did result in logs for you so identifying the cause may be helped along with that additional data) | 22:16 |
| * prometheanfire throws a line out | 22:17 | |
| prometheanfire | https://review.openstack.org/#/c/603544/ clarkb ianw ^ ? | 22:17 |
| ianw | clarkb: has tristanC given you a tour of his ML log analysis tool? | 22:17 |
| notmyname | clarkb: patch set 27 in this file: https://review.openstack.org/#/c/601686/27/.zuul.yaml (the list didn't have a space on lines 245 and 246) | 22:18 |
| clarkb | ianw: I've gotten some tidbits here and there when we end up in the same physical location | 22:18 |
| *** gema has quit IRC | 22:19 | |
| clarkb | notmyname: and you got an error message to show OSA? I"m actually not seeing where the errors was reported. Maybe we don't report it like I thought we did | 22:22 |
| clarkb | ianw: prometheanfire: isn't the pkgmap just a name lookup? we have to add the package to the package list too? | 22:22 |
| prometheanfire | clarkb: down lower in the pkg map I mask it (default is "") | 22:23 |
| clarkb | oh itsalready there just with the not mapped name | 22:23 |
| notmyname | on that one, I'm not sure of what error was reported (or not). corvus also pointed out that run doesn't support a list (yet). | 22:23 |
| ianw | clarkb: it is already listed in https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/infra-package-needs/package-installs.yaml | 22:23 |
| notmyname | clarkb: trying to think if there was a different error we talked to you about | 22:23 |
| clarkb | notmyname: there was one you went to the OSA table in the bar for and it turned out to be a yaml syntax error or ansible syntax error iirc | 22:24 |
| clarkb | I'm assuming it was in that change so looking for that on other patchsets now | 22:25 |
| notmyname | yeah, would have been the same gerrit change | 22:26 |
| clarkb | notmyname: found it http://logs.openstack.org/86/601686/32/check/swift-multinode-rolling-upgrade/4ed9236/job-output.txt.gz#_2018-09-14_19_36_25_291563 thanks | 22:28 |
| notmyname | clarkb: good :-) | 22:29 |
| *** agopi has joined #openstack-infra | 22:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Install gentoolkit on Gentoo https://review.openstack.org/603544 | 22:37 |
| prometheanfire | cool, anyone mind kicking a gentoo build? I can wait as well (D&D tonight) | 22:40 |
| *** rcernin has joined #openstack-infra | 22:44 | |
| *** Tim_ok has quit IRC | 22:53 | |
| *** tpsilva has quit IRC | 22:54 | |
| *** armax has quit IRC | 22:59 | |
| * fungi doesn't have nearly so exciting an evening planned as to include d&d | 23:03 | |
| fungi | prometheanfire: what is it you need done? delete old gentoo images so new ones get built sooner? | 23:04 |
| *** armax has joined #openstack-infra | 23:07 | |
| *** armax has quit IRC | 23:08 | |
| *** bobh has joined #openstack-infra | 23:11 | |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| prometheanfire | fungi: wfm | 23:15 |
| prometheanfire | fungi: how are things on the coast? | 23:15 |
| *** gfidente has quit IRC | 23:15 | |
| *** ansmith has joined #openstack-infra | 23:15 | |
| fungi | still here! happy about that | 23:16 |
| fungi | i'll delete the images in a sec | 23:16 |
| *** jamesmcarthur has quit IRC | 23:19 | |
| *** slaweq has quit IRC | 23:19 | |
| prometheanfire | cool | 23:21 |
| fungi | prometheanfire: new image building gentoo-17-0-systemd-0000000877 on nb01 | 23:27 |
| *** dklyle has quit IRC | 23:29 | |
| fungi | hope your dice remain lucky both on the table and off! | 23:32 |
| * fungi rolls a save vs food coma and consults the relevant chart | 23:33 | |
| * fungi gets back to composing an e-mail abotu e-mail | 23:33 | |
| prometheanfire | cool :D | 23:36 |
| mnaser | yay | 23:36 |
| mnaser | hopefully with ovh back the gate gets churned through overnight | 23:36 |
| openstackgerrit | Merged openstack-infra/git-review master: Clean up vestigal scripting in cmd.py https://review.openstack.org/567297 | 23:43 |
| *** mriedem is now known as mriedem_away | 23:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add notes on manual host configuration runs https://review.openstack.org/516510 | 23:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Use zuul-sphinx README.rst detection https://review.openstack.org/596225 | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
