| *** felipemonteiro_ has joined #openstack-infra | 00:00 | |
| *** agopi has quit IRC | 00:04 | |
| *** harlowja has quit IRC | 00:08 | |
| mnaser | https://review.openstack.org/#/c/560054/ | 00:12 |
|---|---|---|
| mnaser | does anyone know why im getting -- The nodeset "debian-stretch" was not found. | 00:12 |
| mnaser | am i doing something wrong? :> | 00:12 |
| clarkb | mnaser: probably haven't defined the nodeset in openstack-zuul-jobs? | 00:13 |
| clarkb | I think that is where all our nodeests are currently | 00:13 |
| mnaser | i thought i saw it in project-config | 00:13 |
| mnaser | unless being in nodepool != being in zuul as a nodeset | 00:13 |
| *** agopi has joined #openstack-infra | 00:13 | |
| mnaser | https://github.com/openstack-infra/project-config/blob/master/nodepool/nl01.openstack.org.yaml#L16-L17 | 00:13 |
| mnaser | okay yeah it's not ther eyet | 00:13 |
| clarkb | ya thats separate | 00:14 |
| *** zhurong has quit IRC | 00:14 | |
| *** mriedem has joined #openstack-infra | 00:14 | |
| mnaser | pabelanger: are you proposing an update soon to add that? | 00:15 |
| pabelanger | mnaser: Yah, was planning on doing that tomorrow moving, and adding it to zuul-jobs / openstack-zuul-jobs too | 00:16 |
| *** mriedem_afk has quit IRC | 00:16 | |
| pabelanger | mnaser: but if you want to propose, that is oay | 00:16 |
| pabelanger | okay* | 00:16 |
| *** bobh has quit IRC | 00:16 | |
| mnaser | pabelanger: i'll leave it for you, im a bit busy still with other stuff, just wanted to figure out what iw as missing :) | 00:16 |
| pabelanger | ack | 00:16 |
| pabelanger | I'll likely start on it in a few hours | 00:17 |
| *** s-shiono has joined #openstack-infra | 00:20 | |
| *** r-daneel has quit IRC | 00:22 | |
| *** tosky has quit IRC | 00:23 | |
| *** sree has joined #openstack-infra | 00:23 | |
| *** sree has quit IRC | 00:28 | |
| *** olaph1 has joined #openstack-infra | 00:30 | |
| *** olaph has quit IRC | 00:31 | |
| *** HeOS has quit IRC | 00:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-pip master: Add virtualenv option for openstack_pip https://review.openstack.org/560696 | 00:35 |
| ianw | clarkb / pabelanger (you were the last to touch it) : ^ not sure if i'm serious or not. seems to work ... | 00:36 |
| clarkb | ianw: I feel like that might be a in the morning when I don't want to get nerd sniped review | 00:37 |
| *** fkautz has joined #openstack-infra | 00:38 | |
| ianw | clarkb: :) i do get why this becomes a hard puppet problem. in a way, you really want a separate provider for *each* virtualenv. the sort of "global catalog" starts to break down when you start putting the same thing in different containers | 00:40 |
| *** dtruong_ has joined #openstack-infra | 00:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 00:44 |
| *** dtruong has quit IRC | 00:46 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 00:46 |
| *** mpjetta has quit IRC | 00:47 | |
| *** yamamoto has joined #openstack-infra | 00:47 | |
| *** felipemonteiro_ has quit IRC | 00:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-review job https://review.openstack.org/535223 | 00:49 |
| *** mpjetta has joined #openstack-infra | 00:50 | |
| *** odyssey4me has quit IRC | 00:51 | |
| *** odyssey4me has joined #openstack-infra | 00:51 | |
| *** zhangfei has joined #openstack-infra | 00:52 | |
| *** yamamoto has quit IRC | 00:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-upload-to-galaxy job https://review.openstack.org/532084 | 00:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-spec job https://review.openstack.org/532085 | 00:55 |
| *** salv-orlando has joined #openstack-infra | 00:59 | |
| openstackgerrit | Aditya Vaja proposed openstack-infra/project-config master: override existing job config for networking-bigswitch https://review.openstack.org/560698 | 01:01 |
| *** salv-orlando has quit IRC | 01:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 01:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: model: add empty branch attribute to the Tag object https://review.openstack.org/560700 | 01:10 |
| *** aeng has joined #openstack-infra | 01:13 | |
| *** AR2018 has joined #openstack-infra | 01:16 | |
| *** AR2018 has quit IRC | 01:19 | |
| *** iyamahat has quit IRC | 01:24 | |
| *** cshastri has joined #openstack-infra | 01:24 | |
| *** namnh has joined #openstack-infra | 01:25 | |
| *** yamahata has quit IRC | 01:25 | |
| *** shu-mutow-AWAY is now known as shu-mutow | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:32 | |
| shu-mutow | Hi, to run nodejs4-jobs successfully for horizon plugins, they needs to install horizon from master branch. Can we install horizon from master branch into test environment for nodejs4-jobs? | 01:35 |
| *** cshastri has quit IRC | 01:37 | |
| shu-mutow | especially, when horizon plugin uses newly added horizon's function. | 01:38 |
| *** avery has joined #openstack-infra | 01:41 | |
| *** gouthamr has joined #openstack-infra | 01:46 | |
| *** gouthamr has quit IRC | 01:49 | |
| *** gouthamr has joined #openstack-infra | 01:49 | |
| *** markvoelker has quit IRC | 01:52 | |
| *** mriedem has quit IRC | 01:54 | |
| *** stakeda has joined #openstack-infra | 01:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: launcher: handle ZK session loss during handler poll https://review.openstack.org/556335 | 01:56 |
| *** sree has joined #openstack-infra | 01:56 | |
| *** cshastri has joined #openstack-infra | 01:57 | |
| dmsimard | Wow ianw, 50 lines of ruby. I haven't done ruby in so long :/ | 01:57 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 01:57 |
| *** masayukig[m] is now known as masayukig[m]1 | 01:59 | |
| *** salv-orlando has joined #openstack-infra | 02:00 | |
| *** rajinir has quit IRC | 02:00 | |
| *** yamahata has joined #openstack-infra | 02:01 | |
| ianw | dmsimard: yeah, my suggestion is to attempt to keep it that way as long as possible | 02:01 |
| *** masayukig[m]1 is now known as masayukig[m]2 | 02:02 | |
| *** salv-orlando has quit IRC | 02:04 | |
| ianw | "and it is a fluke puppet that it wasn't causing a duplicate resource error." ... a choice cmurphy quote and describes a bit too much of what's going on in our puppet | 02:04 |
| *** sree has quit IRC | 02:05 | |
| *** kmalloc has quit IRC | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:07 | |
| *** edmondsw has joined #openstack-infra | 02:19 | |
| *** masayukig[m]2 is now known as masayuki4 | 02:21 | |
| *** masayuki4 is now known as masayuki6 | 02:22 | |
| *** gongysh has joined #openstack-infra | 02:23 | |
| *** masayuki6 is now known as masayuki | 02:23 | |
| *** avery has left #openstack-infra | 02:23 | |
| *** masayuki is now known as masayuki8 | 02:23 | |
| *** masayuki8 is now known as masayuki | 02:24 | |
| *** edmondsw has quit IRC | 02:24 | |
| *** masayuki is now known as masayuki[m] | 02:24 | |
| *** masayuki[m] is now known as masayukig[m] | 02:24 | |
| *** sshnaidm has quit IRC | 02:26 | |
| wolverineav | clarkb: corvus specifying the specific branch for required-project within the job definition for the EOL-ed branch worked. i added this to the project: https://review.openstack.org/#/c/556657/11/.zuul.yaml | 02:26 |
| wolverineav | adding that for every eol-ed branch seems like a better option than overriding by editing the project template in openstack-infra/project-config. keeps it contained within the project :) | 02:27 |
| wolverineav | thanks clarkb, corvus! | 02:28 |
| *** armaan has quit IRC | 02:29 | |
| *** armaan has joined #openstack-infra | 02:29 | |
| imacdonn | seem that nova-multiattach is failing for all/many cinder jobs ... Apr 12 01:08:11.245108 ubuntu-xenial-ovh-bhs1-0003460059 nova-compute[25976]: INFO nova.virt.libvirt.driver [None req-98b9d947-9721-4543-b105-dd5b78ac04ce None None] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied' | 02:31 |
| *** hongbin has joined #openstack-infra | 02:32 | |
| *** gouthamr has quit IRC | 02:33 | |
| *** wolverineav has quit IRC | 02:34 | |
| *** wolverineav has joined #openstack-infra | 02:34 | |
| *** janki has joined #openstack-infra | 02:35 | |
| *** gouthamr has joined #openstack-infra | 02:35 | |
| *** d0ugal has quit IRC | 02:40 | |
| clarkb | imacdonn: generally that means your user isnt in the right group | 02:40 |
| *** wolverineav has quit IRC | 02:40 | |
| *** gouthamr_ has joined #openstack-infra | 02:40 | |
| clarkb | imacdonn: do you have a link to the failures? and were they working previously? | 02:40 |
| *** gouthamr has quit IRC | 02:40 | |
| imacdonn | clarkb: my user? this is zuul | 02:41 |
| clarkb | imacdonn: the linux user trying ti open libvirt-sock | 02:41 |
| clarkb | its a linux file permissions issue typically due to being in the wrong groul | 02:41 |
| clarkb | *group | 02:41 |
| imacdonn | clarkb: er ok, but this is zuul, not me :) | 02:41 |
| imacdonn | e.g. http://logs.openstack.org/71/560171/2/check/nova-multiattach/d75eb90/ | 02:42 |
| clarkb | right the job is likely broken is my point | 02:42 |
| clarkb | and I'm curious if it ever worked | 02:42 |
| clarkb | zuul only runs what it is told to run | 02:42 |
| imacdonn | I believe that everything was OK until around the time zuul was upgraded earlier today | 02:42 |
| *** jamesmcarthur has joined #openstack-infra | 02:42 | |
| imacdonn | dunno if it's a coincidence | 02:42 |
| *** wolverineav has joined #openstack-infra | 02:44 | |
| *** cshastri has quit IRC | 02:45 | |
| *** d0ugal has joined #openstack-infra | 02:48 | |
| clarkb | it likely is. this error happens deep in devstack | 02:49 |
| clarkb | looks like nova compute fails to start, it should be running as the stack user which should be in the right group but we can dig through the logs to see | 02:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 02:50 |
| *** wolverineav has quit IRC | 02:51 | |
| *** zhangfei has quit IRC | 02:52 | |
| *** markvoelker has joined #openstack-infra | 02:53 | |
| clarkb | imacdonn: http://logs.openstack.org/71/560171/2/check/nova-multiattach/d75eb90/logs/devstacklog.txt.gz#_2018-04-12_01_00_24_784 that is the group created by the package and http://logs.openstack.org/71/560171/2/check/nova-multiattach/d75eb90/logs/devstacklog.txt.gz#_2018-04-12_01_01_07_744 is the group the stack user is added to and they don't match | 02:54 |
| clarkb | imacdonn: I think that is likely the cause of the problem. I'm guessing that packages updated and the group chagned? | 02:54 |
| clarkb | I'm not really in a position to fix that right now but that should hopefully be enough info for someone to work on it if they desire | 02:54 |
| clarkb | (and to confirm I don't think that has anything to do with zuul) | 02:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 02:56 |
| imacdonn | clarkb: Hmm. OK. I'm not sure where all that comes from | 02:56 |
| clarkb | imacdonn: really quickly reading devstack/stackrc the code that selects the group is special cased for multiattach | 02:56 |
| clarkb | I don't understand why | 02:56 |
| clarkb | but that explains why the other jobs aren't broken | 02:57 |
| imacdonn | ok | 02:57 |
| *** olaph has joined #openstack-infra | 02:57 | |
| *** rosmaita has quit IRC | 02:58 | |
| *** olaph1 has quit IRC | 02:59 | |
| *** cshastri has joined #openstack-infra | 02:59 | |
| *** eernst has joined #openstack-infra | 02:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: openstack: convert rate to float https://review.openstack.org/537250 | 02:59 |
| *** salv-orlando has joined #openstack-infra | 03:00 | |
| *** zhangfei has joined #openstack-infra | 03:04 | |
| *** salv-orlando has quit IRC | 03:05 | |
| *** sree has joined #openstack-infra | 03:07 | |
| *** Krenair has quit IRC | 03:08 | |
| *** imacdonn has quit IRC | 03:08 | |
| *** imacdonn has joined #openstack-infra | 03:09 | |
| *** Krenair has joined #openstack-infra | 03:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add emit-job-report role https://review.openstack.org/548425 | 03:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: add variant-description https://review.openstack.org/549748 | 03:11 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** cshastri has quit IRC | 03:14 | |
| *** slaweq has quit IRC | 03:16 | |
| *** VW has joined #openstack-infra | 03:16 | |
| *** germs has quit IRC | 03:16 | |
| *** germs has joined #openstack-infra | 03:17 | |
| *** germs has quit IRC | 03:17 | |
| *** germs has joined #openstack-infra | 03:17 | |
| *** dave-mccowan has quit IRC | 03:18 | |
| *** jchhatbar has joined #openstack-infra | 03:19 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-changes: update for the new api url https://review.openstack.org/557656 | 03:20 |
| *** janki has quit IRC | 03:22 | |
| *** ramineni_ has joined #openstack-infra | 03:24 | |
| *** markvoelker has quit IRC | 03:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] fork puppet-solr https://review.openstack.org/559178 | 03:27 |
| *** jamesmcarthur has quit IRC | 03:28 | |
| ramineni_ | Hi All, recently im seeing failures related to requirements-check job in validating lower constraints | 03:29 |
| ramineni_ | http://logs.openstack.org/84/560284/1/check/requirements-check/d9f02d1/job-output.json.gz | 03:29 |
| *** cshastri has joined #openstack-infra | 03:29 | |
| ramineni_ | do we need to manually update lower-constraints to resolve the failure? | 03:29 |
| imacdonn | ramineni_: rebase should fix it | 03:29 |
| ramineni_ | imacdonn: thanks, is this the issue on infra side? | 03:30 |
| ramineni_ | imacdonn: i ahve seen same failures in multiple patches | 03:30 |
| *** gouthamr_ is now known as gouthamr | 03:31 | |
| imacdonn | ramineni_: http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2018-04-10.log.html#t2018-04-10T21:18:25 | 03:31 |
| *** gyee has quit IRC | 03:33 | |
| *** ramishra has joined #openstack-infra | 03:33 | |
| imacdonn | ramineni_: your failure looks slightly different, but I bet it's related | 03:34 |
| *** armaan has quit IRC | 03:34 | |
| *** dhajare has joined #openstack-infra | 03:34 | |
| *** armaan has joined #openstack-infra | 03:35 | |
| ramineni_ | imacdonn: ya , and also tox change is already there long time back, so my patch is rebased on that , but still seeing failures | 03:35 |
| *** armaan has quit IRC | 03:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object https://review.openstack.org/560700 | 03:39 |
| *** d0ugal has quit IRC | 03:39 | |
| clarkb | you need to increase the test-requirement.txt version of that package to >=0.5 to match the lower constraint value | 03:39 |
| *** armaan has joined #openstack-infra | 03:39 | |
| clarkb | it isnt an infra problem. The job is catching that lower constraints is higher than the lowest version allowed in test requirements | 03:39 |
| *** psachin has joined #openstack-infra | 03:40 | |
| ramineni_ | clarkb: oh, doesnt test-requirements.txt get automatically updated, manual update is required everytime? | 03:41 |
| clarkb | ramineni_: there were recent changes to not update it as aggressively aiui | 03:41 |
| clarkb | but I'd have to read the mailing list to get all the details straight | 03:42 |
| ramineni_ | clarkb: ok, thanks , one more question , i see in some places tehre is mismatch in lower-constraints in the project and that of requirements project , do we need to update them also from time to time | 03:46 |
| *** armaan has quit IRC | 03:47 | |
| *** gongysh has quit IRC | 03:48 | |
| *** d0ugal has joined #openstack-infra | 03:48 | |
| *** nicolasbock has quit IRC | 03:49 | |
| *** cshastri has quit IRC | 03:52 | |
| clarkb | ramineni_: again I'm not sure of all the details. There was a large mailing list thread about it that should ave the answers | 03:53 |
| clarkb | also you can ask the requirements team on their irc channel | 03:53 |
| ramineni_ | clarkb: sure, thanks for the pointers | 03:53 |
| *** dhajare has quit IRC | 03:56 | |
| prometheanfire | clarkb: pabelanger: looks like we have the new dib release, next steps are to just start using the gentoo image then? | 03:57 |
| *** VW has quit IRC | 03:58 | |
| *** salv-orlando has joined #openstack-infra | 04:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Minor fork puppet-solr https://review.openstack.org/559178 | 04:03 |
| *** udesale has joined #openstack-infra | 04:04 | |
| *** rlandy has quit IRC | 04:04 | |
| *** salv-orlando has quit IRC | 04:05 | |
| *** cshastri has joined #openstack-infra | 04:05 | |
| *** edmondsw has joined #openstack-infra | 04:07 | |
| *** gouthamr has quit IRC | 04:10 | |
| *** edmondsw has quit IRC | 04:12 | |
| *** markvoelker has joined #openstack-infra | 04:23 | |
| *** olaph1 has joined #openstack-infra | 04:23 | |
| *** olaph has quit IRC | 04:24 | |
| *** claudiub has joined #openstack-infra | 04:26 | |
| *** markvoelker has quit IRC | 04:27 | |
| openstackgerrit | Merged openstack-infra/zuul master: Report to all reporters even if one fails https://review.openstack.org/557859 | 04:29 |
| openstackgerrit | Merged openstack-infra/zuul master: Reorganize "Zuul From Scratch" document https://review.openstack.org/556988 | 04:29 |
| *** dhajare has joined #openstack-infra | 04:34 | |
| *** tdasilva has quit IRC | 04:36 | |
| *** markvoelker has joined #openstack-infra | 04:38 | |
| *** hongbin has quit IRC | 04:38 | |
| *** claudiub|2 has joined #openstack-infra | 04:42 | |
| *** olaph1 has quit IRC | 04:42 | |
| *** olaph has joined #openstack-infra | 04:42 | |
| *** kjackal has joined #openstack-infra | 04:43 | |
| *** claudiub has quit IRC | 04:45 | |
| *** hongbin has joined #openstack-infra | 04:46 | |
| *** hongbin has quit IRC | 04:51 | |
| *** iyamahat has joined #openstack-infra | 04:53 | |
| *** tdasilva has joined #openstack-infra | 04:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove castellan legacy jobs https://review.openstack.org/551008 | 04:57 |
| *** lpetrut has joined #openstack-infra | 05:02 | |
| *** salv-orlando has joined #openstack-infra | 05:02 | |
| *** salv-orlando has quit IRC | 05:05 | |
| *** salv-orlando has joined #openstack-infra | 05:05 | |
| *** chandankumar has quit IRC | 05:06 | |
| *** chandankumar has joined #openstack-infra | 05:07 | |
| *** links has joined #openstack-infra | 05:09 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Allow installing glean from package https://review.openstack.org/557867 | 05:10 |
| *** iyamahat has quit IRC | 05:11 | |
| *** gongysh has joined #openstack-infra | 05:11 | |
| *** iyamahat has joined #openstack-infra | 05:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove castellan legacy jobs https://review.openstack.org/551011 | 05:15 |
| *** jtomasek has joined #openstack-infra | 05:18 | |
| *** ianychoi_ has joined #openstack-infra | 05:19 | |
| *** jbadiapa has quit IRC | 05:19 | |
| *** ianychoi has quit IRC | 05:23 | |
| *** jtomasek has quit IRC | 05:24 | |
| *** jaosorior has quit IRC | 05:26 | |
| *** hamzy has joined #openstack-infra | 05:27 | |
| *** kjackal has quit IRC | 05:27 | |
| *** kjackal has joined #openstack-infra | 05:27 | |
| *** markvoelker has quit IRC | 05:29 | |
| *** markvoelker has joined #openstack-infra | 05:30 | |
| *** eernst has quit IRC | 05:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: gerritbot: Announce stable branch changes of openstack/requirements https://review.openstack.org/560677 | 05:30 |
| *** markvoelker has quit IRC | 05:34 | |
| *** d0ugal has quit IRC | 05:41 | |
| *** lpetrut has quit IRC | 05:46 | |
| *** d0ugal has joined #openstack-infra | 05:48 | |
| *** openstackgerrit has quit IRC | 05:48 | |
| *** iranzo has joined #openstack-infra | 05:49 | |
| *** olaph has quit IRC | 05:52 | |
| *** olaph has joined #openstack-infra | 05:52 | |
| *** dhajare has quit IRC | 05:54 | |
| *** hrubi has quit IRC | 05:55 | |
| *** jaosorior has joined #openstack-infra | 05:55 | |
| *** lpetrut has joined #openstack-infra | 05:55 | |
| *** edmondsw has joined #openstack-infra | 05:56 | |
| *** udesale_ has joined #openstack-infra | 05:56 | |
| *** neiloy_ has joined #openstack-infra | 05:56 | |
| *** udesale has quit IRC | 05:59 | |
| *** edmondsw has quit IRC | 06:00 | |
| *** openstackgerrit has joined #openstack-infra | 06:02 | |
| openstackgerrit | Merged openstack-infra/nodepool master: openstack: convert rate to float https://review.openstack.org/537250 | 06:02 |
| *** dhajare has joined #openstack-infra | 06:03 | |
| *** threestrands has joined #openstack-infra | 06:05 | |
| *** threestrands has quit IRC | 06:05 | |
| *** threestrands has joined #openstack-infra | 06:05 | |
| *** iranzo has quit IRC | 06:09 | |
| *** markvoelker has joined #openstack-infra | 06:11 | |
| *** e0ne has joined #openstack-infra | 06:11 | |
| *** alex_xu has quit IRC | 06:11 | |
| *** markvoelker has quit IRC | 06:16 | |
| *** alex_xu has joined #openstack-infra | 06:16 | |
| *** germs has quit IRC | 06:18 | |
| *** germs has joined #openstack-infra | 06:18 | |
| *** germs has quit IRC | 06:18 | |
| *** germs has joined #openstack-infra | 06:18 | |
| *** neiloy_ has quit IRC | 06:19 | |
| *** Hal1 has quit IRC | 06:20 | |
| *** aeng has quit IRC | 06:21 | |
| *** sfeole has quit IRC | 06:23 | |
| *** dbecker has joined #openstack-infra | 06:25 | |
| *** andreas_s has joined #openstack-infra | 06:26 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove castellan legacy jobs https://review.openstack.org/551011 | 06:26 |
| *** dingyichen has joined #openstack-infra | 06:27 | |
| frickler | clarkb: imacdonn: https://review.openstack.org/554314 was merged earlier, this seems to affect the multiattach jobs via an updated libvirt. I was already sceptical about that patch, we may need a revert if the issue persists | 06:30 |
| *** jtomasek has joined #openstack-infra | 06:31 | |
| *** sfeole has joined #openstack-infra | 06:31 | |
| *** lpetrut has quit IRC | 06:33 | |
| *** salv-orlando has quit IRC | 06:35 | |
| *** salv-orlando has joined #openstack-infra | 06:35 | |
| *** iranzo has joined #openstack-infra | 06:35 | |
| *** iranzo has quit IRC | 06:35 | |
| *** iranzo has joined #openstack-infra | 06:35 | |
| *** armaan has joined #openstack-infra | 06:36 | |
| *** liujiong has joined #openstack-infra | 06:37 | |
| *** vivsoni has quit IRC | 06:38 | |
| *** ramishra has quit IRC | 06:39 | |
| *** salv-orlando has quit IRC | 06:39 | |
| *** salv-orlando has joined #openstack-infra | 06:42 | |
| *** caphrim007 has quit IRC | 06:43 | |
| *** caphrim007 has joined #openstack-infra | 06:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: build opensuse images from the openstack mirror https://review.openstack.org/560662 | 06:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable voting Python 3 unit tests for monasca-ui https://review.openstack.org/560323 | 06:44 |
| *** aojea has joined #openstack-infra | 06:47 | |
| *** hashar has joined #openstack-infra | 06:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix implicit role for repos named ansible https://review.openstack.org/560554 | 06:52 |
| *** agopi has quit IRC | 06:53 | |
| *** alexchadin has joined #openstack-infra | 06:53 | |
| *** e0ne has quit IRC | 06:58 | |
| *** armaan_ has joined #openstack-infra | 06:59 | |
| *** armaan has quit IRC | 07:00 | |
| *** ekhugen has quit IRC | 07:03 | |
| *** ekhugen has joined #openstack-infra | 07:04 | |
| *** armaan_ has quit IRC | 07:04 | |
| *** hemna_ has quit IRC | 07:04 | |
| *** yamamoto has joined #openstack-infra | 07:05 | |
| *** e0ne has joined #openstack-infra | 07:05 | |
| *** armaan has joined #openstack-infra | 07:08 | |
| *** kjackal_ has joined #openstack-infra | 07:08 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** kjackal has quit IRC | 07:09 | |
| *** ramishra has joined #openstack-infra | 07:13 | |
| *** markvoelker has joined #openstack-infra | 07:15 | |
| *** ralonsoh has joined #openstack-infra | 07:15 | |
| *** gfidente has joined #openstack-infra | 07:17 | |
| *** gfidente has joined #openstack-infra | 07:17 | |
| *** olaph1 has joined #openstack-infra | 07:18 | |
| *** olaph has quit IRC | 07:19 | |
| *** zoli is now known as zoli|wfh | 07:20 | |
| *** zoli|wfh is now known as zoli | 07:20 | |
| *** florianf has joined #openstack-infra | 07:23 | |
| *** electrofelix has joined #openstack-infra | 07:24 | |
| *** jbadiapa has joined #openstack-infra | 07:28 | |
| *** udesale__ has joined #openstack-infra | 07:28 | |
| *** xinliang has quit IRC | 07:30 | |
| *** udesale_ has quit IRC | 07:31 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** liujiong has quit IRC | 07:34 | |
| *** liujiong has joined #openstack-infra | 07:34 | |
| *** xinliang has joined #openstack-infra | 07:35 | |
| *** armaan has quit IRC | 07:35 | |
| *** armaan has joined #openstack-infra | 07:35 | |
| *** tosky has joined #openstack-infra | 07:38 | |
| *** tesseract has joined #openstack-infra | 07:39 | |
| *** armaan has quit IRC | 07:40 | |
| *** edmondsw has joined #openstack-infra | 07:44 | |
| *** jpich has joined #openstack-infra | 07:45 | |
| *** edmondsw has quit IRC | 07:48 | |
| *** dbecker has quit IRC | 07:51 | |
| *** sshnaidm has joined #openstack-infra | 07:52 | |
| *** e0ne_ has joined #openstack-infra | 07:52 | |
| *** e0ne__ has joined #openstack-infra | 07:53 | |
| *** e0ne has quit IRC | 07:53 | |
| *** lucas-afk is now known as lucasagomes | 07:55 | |
| *** frickler has quit IRC | 07:56 | |
| *** e0ne_ has quit IRC | 07:57 | |
| *** gcb has quit IRC | 07:57 | |
| *** Hal has joined #openstack-infra | 07:57 | |
| *** apetrich has quit IRC | 07:59 | |
| *** armaan has joined #openstack-infra | 08:00 | |
| *** gongysh has quit IRC | 08:01 | |
| *** frickler has joined #openstack-infra | 08:03 | |
| *** adreznec has quit IRC | 08:05 | |
| *** adreznec has joined #openstack-infra | 08:06 | |
| *** masayukig has quit IRC | 08:12 | |
| *** pcaruana has joined #openstack-infra | 08:12 | |
| *** masayukig[m] is now known as masayukig | 08:12 | |
| *** masayukig_ has joined #openstack-infra | 08:13 | |
| *** d0ugal has quit IRC | 08:14 | |
| *** armaan has quit IRC | 08:15 | |
| *** armaan has joined #openstack-infra | 08:15 | |
| xinliang | pabelanger, ianw: From the log of https://review.openstack.org/#/c/557659/, it seems that debian node mirror configuration is not work | 08:16 |
| *** yamamoto has quit IRC | 08:17 | |
| *** d0ugal has joined #openstack-infra | 08:17 | |
| xinliang | when installing libffi-dev pkg it said "broken package ", but I've tried the mirror on my locall machine, it can install libffi-dev successfully | 08:17 |
| *** dsariel has joined #openstack-infra | 08:18 | |
| xinliang | no idea what thing is going wrong | 08:18 |
| *** armaan has quit IRC | 08:20 | |
| *** gongysh has joined #openstack-infra | 08:23 | |
| *** masayukig is now known as masayukig[m] | 08:25 | |
| ianw | xinliang: which log is that in? | 08:27 |
| *** masayukig_ is now known as masayukig | 08:27 | |
| ianw | oh i see ... Unable to correct problems, you have held broken packages | 08:27 |
| ianw | hmm | 08:27 |
| xinliang | this one: http://logs.openstack.org/59/557659/10/experimental/kolla-build-debian-source-arm64/0ce7753/job-output.txt.gz#_2018-04-10_11_08_16_088178 | 08:27 |
| xinliang | ianw: yes | 08:28 |
| *** s-shiono has quit IRC | 08:29 | |
| xinliang | ianw: is there a way to debug that debian image? I mean use that image to luanch an vm and see what's the problem? | 08:29 |
| ianw | $ ssh root@211.148.24.206 | 08:30 |
| ianw | root@211.148.24.206's password: | 08:30 |
| *** salv-orlando has quit IRC | 08:30 | |
| ianw | xinliang: ^ yeah ... not really :/ i can't log into it ... | 08:31 |
| *** salv-orlando has joined #openstack-infra | 08:31 | |
| ianw | which suggests glean isn't rolling out the ssh auth correctly | 08:31 |
| ianw | but ... zuul got in? | 08:31 |
| *** yamamoto has joined #openstack-infra | 08:32 | |
| *** lpetrut has joined #openstack-infra | 08:33 | |
| xinliang | ianw: I use that debian image to create an vm with my key , but I can't ssh into it . cloud-init seems not be installed? | 08:33 |
| ianw | xinliang: we don't use cloud-init, we use simple-init & glean, but same sort of thing. i should get the info from configdrive | 08:34 |
| *** efoley has joined #openstack-infra | 08:34 | |
| xinliang | ianw: so the only way is get the priv key and ssh with zuul user, right | 08:35 |
| *** salv-orlando has quit IRC | 08:35 | |
| *** masayukig is now known as masayukig_ | 08:35 | |
| *** masayukig[m] is now known as masayukig | 08:36 | |
| *** yamamoto has quit IRC | 08:36 | |
| ianw | xinliang: yeah, i can get in via that | 08:39 |
| *** masayukig_ has quit IRC | 08:40 | |
| xinliang | ok, please help to take a look if you have time, thanks | 08:40 |
| ianw | ok, blkid is not showing /dev/sr0 | 08:44 |
| ianw | which i think means the config-drive is not there, which explains why no other keys are rolled out | 08:44 |
| *** sambetts|afk is now known as sambetts | 08:45 | |
| *** dingyichen has quit IRC | 08:45 | |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** apetrich has joined #openstack-infra | 08:49 | |
| *** linkmark has joined #openstack-infra | 08:53 | |
| *** armaan has joined #openstack-infra | 08:53 | |
| *** armaan_ has joined #openstack-infra | 08:54 | |
| *** armaan has quit IRC | 08:57 | |
| *** dpawlik has quit IRC | 09:02 | |
| *** dpawlik has joined #openstack-infra | 09:08 | |
| *** zhipeng has joined #openstack-infra | 09:10 | |
| ianw | xinliang / pabelanger : ok, i'm out of time for now but this is the root problem. i've tried rebooting it with cdrom and sr_mod in /etc/modules, but still doesn't show up | 09:12 |
| xinliang | ianw: ok | 09:13 |
| ianw | xinliang: i remember seeing this with xenial; and switching to the hwe kernel made it "just work" | 09:15 |
| ianw | hrw / gema : ^ any ideas on missing cd-rom devices on debian? | 09:16 |
| *** iyamahat has quit IRC | 09:17 | |
| frickler | ianw: I think it will need a special virt-* driver, can try to look into it later | 09:19 |
| ianw | frickler: yeah, i'm not so sure; it works on ubuntu | 09:21 |
| xinliang | ianw: trying to understand this issue, you mean config-drive need a cd-rom device which is missing on debian. This is the root cuase mirror is not working? | 09:26 |
| ianw | xinliang: no, config-drive appears as an attached cdrom drive, and debian is not seeing it at all. that drive contains config info such as the ssh-keys to apply | 09:27 |
| ianw | not really to do with the mirror, but it's quite hard to debug that until we can log into it | 09:28 |
| ianw | i feel like the sym53c8xx module is involved somehow here | 09:28 |
| xinliang | ianw: i see | 09:28 |
| ianw | ubuntu seems to have it, debian not | 09:28 |
| *** dbecker has joined #openstack-infra | 09:31 | |
| xinliang | ianw: one thins stange is that deiban installed backport kernel deb. which I think it should install non-backport one | 09:31 |
| xinliang | s/thins/thing/ | 09:31 |
| *** salv-orlando has joined #openstack-infra | 09:31 | |
| xinliang | any other apt preference rule putting into debian when buidling debian image? | 09:32 |
| ianw | xinliang: yeah, maybe we need to actually set a lower preference on that? | 09:35 |
| *** sree has quit IRC | 09:36 | |
| *** salv-orlando has quit IRC | 09:37 | |
| *** salv-orlando has joined #openstack-infra | 09:39 | |
| ianw | xinliang: i think this is definitely related to missing drivers in debian : config.common.amd64:CONFIG_SCSI_SYM53C8XX_2=m | 09:43 |
| ianw | that's on ubuntu, i can't find it on debian | 09:43 |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| *** alexchadin has quit IRC | 09:47 | |
| *** shu-mutow is now known as shu-mutow-AWAY | 09:48 | |
| *** yamamoto has quit IRC | 09:48 | |
| xinliang | yes, I see CONFIG_SCSI_SYM53C8XX_2 is not set on debian | 09:48 |
| ianw | xinliang: yeah, 00:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a | 09:48 |
| ianw | from lspci | 09:49 |
| *** yamamoto has joined #openstack-infra | 09:49 | |
| *** armaan_ has quit IRC | 09:49 | |
| *** alexchadin has joined #openstack-infra | 09:52 | |
| *** yamamoto has quit IRC | 09:52 | |
| *** yamamoto has joined #openstack-infra | 09:54 | |
| *** HeOS has joined #openstack-infra | 09:54 | |
| *** Hal has quit IRC | 09:55 | |
| *** stakeda has quit IRC | 09:57 | |
| *** panda|off is now known as panda | 09:57 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** kmalloc has joined #openstack-infra | 10:01 | |
| *** yamamoto has joined #openstack-infra | 10:01 | |
| *** hrw has joined #openstack-infra | 10:02 | |
| hrw | hi | 10:02 |
| hrw | xinliang: so how config-drive is described? | 10:03 |
| hrw | ianw: please use virtio-blk/virtio-scsi instead of real scsi controllers emulation | 10:04 |
| xinliang | hrw, i don't know, ianw please explain about this | 10:04 |
| hrw | ianw: aarch64 VM instances are supposed to be 'modern world' instead of 'anything from 90s' ;D | 10:05 |
| *** namnh has quit IRC | 10:06 | |
| ianw | hrw: well ... i'm all ears on how to get the cdrom seen? | 10:06 |
| hrw | ianw: virtio-scsi cdrom | 10:07 |
| ianw | is this something under my control? it's coming from nova | 10:07 |
| ianw | which i presume is setting this all up under the hood with qemu/kvm flags of intense complexity | 10:08 |
| hrw | nova passes cdrom to libvirt which should use sane values | 10:08 |
| hrw | ianw: can you get that instance xml? | 10:08 |
| ianw | hrw: well, it would seem maybe it isn't :) | 10:08 |
| ianw | i can't, it's on the linaro cloud | 10:08 |
| hrw | ah. ok | 10:08 |
| *** liujiong has quit IRC | 10:08 | |
| ianw | i still think it wouldn't *hurt* for debian to enable the module, to avoid this sort of thing for unsuspecting installers | 10:09 |
| ianw | but if we can reconfigure the cloud to do it better, great | 10:09 |
| hrw | ianw: and then piix3-ata? | 10:09 |
| hrw | ianw: let me check on my cloud setup how cdrom is handled | 10:09 |
| *** pbourke has quit IRC | 10:14 | |
| *** pbourke has joined #openstack-infra | 10:15 | |
| *** alexchadin has quit IRC | 10:15 | |
| *** owalsh_afk is now known as owalsh | 10:19 | |
| ianw | hrw / xinliang : i filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895532 anyway. i'm sure if people have opinions, they will make it into that :) | 10:20 |
| openstack | Debian bug 895532 in src:linux "linux-image-4.14.0-0.bpo.3-arm64: Enable CONFIG_SCSI_SYM53C8XX_2 for ARM64" [Normal,Open] | 10:20 |
| hrw | sure | 10:20 |
| *** zoli is now known as zoli|lunch | 10:20 | |
| *** lpetrut has quit IRC | 10:21 | |
| hrw | <controller type='scsi' index='0' model='lsilogic'> | 10:23 |
| hrw | ok, I see the problem | 10:23 |
| hrw | ianw: please add hw_scsi_mode=virtio-scsi to the image and recreate vm | 10:24 |
| hrw | ianw: erm. hw_scsi_model | 10:24 |
| *** threestrands has quit IRC | 10:24 | |
| ianw | hrw: hmm, that's already set -> https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nb03.openstack.org.yaml#n35 | 10:26 |
| hrw | @$#@# | 10:27 |
| hrw | ianw: ok, will take a look where nova/libvirt does it and inform you back. | 10:28 |
| hrw | ianw: so you can have rest of evening for you instead of fighting issues | 10:28 |
| hrw | but looking at nova source it looks like it uses lsilogic on vmware only. | 10:29 |
| hrw | ianw: I have set hw_scsi_model to virtio-scsi locally and created VM instanse has proper scsi controller | 10:31 |
| ianw | hmm, i'm fairly sure that nodepool via shade is setting those tags with glance correctly | 10:32 |
| ianw | that really leaves something inside the linaro cloud not obeying it? | 10:33 |
| hrw | xinliang: you run queens-2018* images in CN cloud, right? | 10:33 |
| xinliang | hrw: not me | 10:34 |
| hrw | xinliang: ok, can you check what exactly we run there? | 10:35 |
| xinliang | hrw: not understand what you mean? you mean the ci job which I debugged running queens-2018* kolla image on CN cloud? | 10:37 |
| ianw | just for sanity, confirmed the tags are in glance http://paste.openstack.org/show/719029/ | 10:37 |
| hrw | xinliang: nevermind. | 10:37 |
| hrw | ianw: thx | 10:39 |
| hrw | ianw: bare metal runs ubuntu xenial? | 10:39 |
| ianw | hrw: not sure tbh. i only have access as an openstack user | 10:40 |
| hrw | ianw: uname -a? lsb_release -a? | 10:42 |
| hrw | ianw: virsh --version? | 10:42 |
| *** apetrich has quit IRC | 10:44 | |
| *** neiloy_ has joined #openstack-infra | 10:44 | |
| ianw | hrw: i can't do any of that, we only have api credentials in the linaro cloud. i think we need gema or someone who can log in | 10:44 |
| *** wznoinsk_ is now known as wznoinsk | 10:44 | |
| hrw | ok, I will handle that | 10:44 |
| *** e0ne__ has quit IRC | 10:46 | |
| *** olaph has joined #openstack-infra | 10:49 | |
| *** olaph1 has quit IRC | 10:50 | |
| gema | hrw: bare metal runs ERP | 10:51 |
| gema | hrw: debian | 10:51 |
| hrw | gema: thx | 10:51 |
| gema | np | 10:51 |
| gema | hrw: cn1 has not been updated to containers yet, we are running newton erp packages | 10:51 |
| hrw | gema: o fu.. | 10:51 |
| gema | hrw: the first queens cloud is UK | 10:51 |
| gema | hrw: and it is not open for business yet | 10:52 |
| gema | still testing it | 10:52 |
| *** dhill_ has quit IRC | 10:52 | |
| gema | hrw: once that is solid, we'll upgrade the rest | 10:52 |
| *** dhill_ has joined #openstack-infra | 10:53 | |
| gema | hrw: the network link in cn1 is being changed to a faster one, afaik the cable is there, the connectivity is faster, we just need to move the users to the new link | 10:53 |
| *** efoley has quit IRC | 10:54 | |
| gema | i.e. ips change | 10:54 |
| hrw | gema: I managed to forget how libvirt 2.2.10 works | 10:54 |
| gema | hrw: badly :P | 10:54 |
| *** odyssey4me has quit IRC | 10:55 | |
| *** odyssey4me has joined #openstack-infra | 10:55 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** sshnaidm is now known as sshnaidm|lnch | 10:58 | |
| gema | for us anyway | 10:59 |
| hrw | but even with such old version it should use hw_scsi_model one ;( | 10:59 |
| gema | hrw: we backported your change to cn1 | 10:59 |
| gema | that works | 10:59 |
| gema | ah, no, scsi model no | 10:59 |
| gema | we backported the image type one | 10:59 |
| hrw | gema: infra sets hw_firmware_type anyway | 11:00 |
| *** odyssey4me has quit IRC | 11:00 | |
| gema | ah, ok | 11:00 |
| *** odyssey4me has joined #openstack-infra | 11:00 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** tpsilva has joined #openstack-infra | 11:05 | |
| *** gongysh has quit IRC | 11:06 | |
| hrw | DC-339 created | 11:07 |
| *** yamamoto has joined #openstack-infra | 11:08 | |
| *** neiloy_ has quit IRC | 11:10 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** zhangfei has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:11 | |
| hrw | ops | 11:13 |
| hrw | gema: should I reopen #38 issue to get 'j12-m1-c03n1 console - mlab-cb - mr provisioner' as <title> or do you want new issue? | 11:18 |
| *** edmondsw has joined #openstack-infra | 11:20 | |
| *** sree has joined #openstack-infra | 11:20 | |
| *** zoli|lunch is now known as zoli | 11:22 | |
| *** zoli is now known as zoli|wfh | 11:22 | |
| *** zoli|wfh is now known as zoli | 11:22 | |
| *** owalsh is now known as owalsh_afk | 11:23 | |
| * hrw uses far too many windows | 11:24 | |
| *** edmondsw has quit IRC | 11:24 | |
| *** sree has quit IRC | 11:24 | |
| *** lpetrut has joined #openstack-infra | 11:25 | |
| *** nicolasbock has joined #openstack-infra | 11:30 | |
| *** lucasagomes is now known as lucas-hungry | 11:31 | |
| *** yamamoto has quit IRC | 11:33 | |
| *** derekh_ has joined #openstack-infra | 11:34 | |
| *** rcernin has quit IRC | 11:35 | |
| *** derekh has quit IRC | 11:36 | |
| *** IdoKaplan has joined #openstack-infra | 11:43 | |
| IdoKaplan | Hi, Is anyone here can help me please with Manila? I'm getting an error when I'm trying to add "user" access to "cifs" protocol - "ERROR oslo_messaging.rpc.server InvalidShareAccess: Invalid access rule: Only 'ip' access type allowed" . Please advise | 11:43 |
| *** cshastri has quit IRC | 11:45 | |
| *** ldnunes has joined #openstack-infra | 11:46 | |
| *** ramineni_ has left #openstack-infra | 11:47 | |
| tbarron | IdoKaplan: suggest you take the question to #openstack-manila | 11:51 |
| *** jpena is now known as jpena|lunch | 11:53 | |
| *** amoralej is now known as amoralej|lunch | 11:53 | |
| *** IdoKaplan has left #openstack-infra | 11:55 | |
| *** apetrich has joined #openstack-infra | 12:01 | |
| *** rfolco|off is now known as rfolco|rover | 12:02 | |
| *** ralonsoh_ has joined #openstack-infra | 12:02 | |
| *** udesale__ has quit IRC | 12:05 | |
| *** apetrich has quit IRC | 12:06 | |
| *** apetrich has joined #openstack-infra | 12:06 | |
| *** ralonsoh has quit IRC | 12:06 | |
| *** olaph1 has joined #openstack-infra | 12:13 | |
| *** olaph has quit IRC | 12:13 | |
| *** rosmaita has joined #openstack-infra | 12:17 | |
| *** r-daneel has joined #openstack-infra | 12:18 | |
| *** eharney has joined #openstack-infra | 12:21 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| pabelanger | morning | 12:29 |
| pabelanger | prometheanfire: yah, if we tagged a new DIB we can unpause gentoo and nodepool-builders should pick it up | 12:30 |
| gema | hrw: new issue, please | 12:30 |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| *** sshnaidm|lnch is now known as sshnaidm | 12:32 | |
| *** dave-mccowan has joined #openstack-infra | 12:32 | |
| *** rfolco|rover is now known as rfolco|notruck | 12:34 | |
| *** rfolco|notruck is now known as rfolco|free | 12:34 | |
| *** rfolco|free is now known as rfolco | 12:34 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** dayou has quit IRC | 12:35 | |
| *** trown|outtypewww is now known as trown|brb | 12:37 | |
| *** dpawlik has quit IRC | 12:37 | |
| *** dayou has joined #openstack-infra | 12:37 | |
| *** trown|brb is now known as trown | 12:37 | |
| pabelanger | dmsimard: clarkb: fungi: I'm getting bounced emails for etherpad01.o.o, but don't see any DNS entries. | 12:38 |
| fungi | pabelanger: that's probably why | 12:39 |
| dmsimard | That's from my 16.04 attempt | 12:39 |
| fungi | should either delete that instance for now or add dns entries for it | 12:39 |
| dmsimard | I'll tear it down | 12:39 |
| pabelanger | +1 | 12:39 |
| pabelanger | thanks | 12:39 |
| *** iranzo has quit IRC | 12:44 | |
| *** HeOS has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:47 | |
| *** kgiusti has joined #openstack-infra | 12:49 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** dklyle has quit IRC | 12:51 | |
| pabelanger | hrw: gema: Is there any specs on linaro cloud in cn1? eg: hardware info / vendor | 12:52 |
| hrw | pabelanger: misc aarch64 servers with 32+GB of ram and some GB/TB of storage. different vendors. That's probably the closest to what I can say | 12:54 |
| hrw | pabelanger: some machines may be under NDA, some never reached mass market, some are mass market available, some are no longer mass market available. | 12:55 |
| pabelanger | hrw: prefect, we had a question on the infra ML, and wanted to share some public information | 12:56 |
| pabelanger | thanks | 12:56 |
| hrw | pabelanger: which thread? | 12:56 |
| *** dklyle has joined #openstack-infra | 12:57 | |
| *** iranzo has joined #openstack-infra | 12:59 | |
| *** iranzo has quit IRC | 12:59 | |
| *** iranzo has joined #openstack-infra | 12:59 | |
| *** sree has joined #openstack-infra | 13:00 | |
| *** jbadiapa has quit IRC | 13:00 | |
| *** dpawlik has joined #openstack-infra | 13:00 | |
| *** tobberydberg_ is now known as tobberydberg | 13:01 | |
| *** edmondsw has joined #openstack-infra | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| pabelanger | http://lists.openstack.org/pipermail/openstack-infra/2018-April/005921.html | 13:01 |
| *** yamamoto has quit IRC | 13:01 | |
| pabelanger | but, that seems to be back in Jan | 13:02 |
| pabelanger | unsure why my MTA decided to move it into my inbox | 13:02 |
| *** bobh has joined #openstack-infra | 13:02 | |
| hrw | thx | 13:02 |
| hrw | ;D | 13:03 |
| *** belliott has joined #openstack-infra | 13:03 | |
| *** ralonsoh__ has joined #openstack-infra | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** lucas-hungry is now known as lucasagomes | 13:05 | |
| *** Hal has joined #openstack-infra | 13:05 | |
| gema | pabelanger: what is OSP CI? | 13:05 |
| *** ralonsoh_ has quit IRC | 13:05 | |
| *** kashyap has joined #openstack-infra | 13:06 | |
| kashyap | clarkb: Just keeping posted, this is now merged in master and also Queens and Pike: https://git.openstack.org/cgit/openstack/nova/commit/?h=master&id=6b601b7 ("libvirt: Allow to specify granular CPU feature flags") | 13:06 |
| kashyap | clarkb: Just to recap: That is to alleviate the Nova instance performance degradation as a result of applying "Meltdown" CVE fixes. | 13:07 |
| pabelanger | gema: downstream product (openstack) CI at RedHat, so a little curious why it was posted on openstack-infra ML | 13:07 |
| * kashyap will write a blog tomorrow evening | 13:07 | |
| *** jcoufal has joined #openstack-infra | 13:07 | |
| *** jchhatbar is now known as janki | 13:07 | |
| gema | pabelanger: ah, ok | 13:07 |
| gema | pabelanger: they should have Arm64 HW internally, no? | 13:07 |
| *** panda is now known as panda|rover | 13:08 | |
| pabelanger | gema: I'm unsure if they are looking for hardware for arm64 or wanting to use an arm64 cloud provider to test things | 13:08 |
| pabelanger | gema: not sure the status of it honestly, I would imagine tonyb knows more | 13:08 |
| *** yamamoto has quit IRC | 13:09 | |
| gema | pabelanger: ack, will ask around, thanks for bring that up to the top of my inbox :D | 13:09 |
| *** dklyle has quit IRC | 13:09 | |
| *** Goneri has joined #openstack-infra | 13:14 | |
| *** jbadiapa has joined #openstack-infra | 13:15 | |
| *** hrubi has joined #openstack-infra | 13:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add debian-stable nodeset https://review.openstack.org/560897 | 13:18 |
| pabelanger | clarkb: fungi: mnaser: ^I decided to create debian-stable nodeset for debian-stretch. Mostly to aid the transition process when debian buster comes online. This comes from the idea of creating fedora-latest to help avoid updating intree job nodesets when we rotate out non ubuntu / centos DIBs. | 13:20 |
| *** edmondsw has quit IRC | 13:20 | |
| *** yamamoto has joined #openstack-infra | 13:21 | |
| pabelanger | but also happy to just do debian-stretch nodeset | 13:21 |
| *** edmondsw has joined #openstack-infra | 13:21 | |
| *** edmondsw has quit IRC | 13:21 | |
| *** dhill_ has quit IRC | 13:24 | |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| *** dhill_ has joined #openstack-infra | 13:24 | |
| *** yamamoto has quit IRC | 13:24 | |
| *** andreas_s has quit IRC | 13:27 | |
| odyssey4me | hi everyone, I have two new repositories which have 2x+2 but no +w - any chance we get can get these processed today? https://review.openstack.org/560058 & https://review.openstack.org/560061 | 13:27 |
| *** links has quit IRC | 13:27 | |
| *** jaosorior has quit IRC | 13:29 | |
| *** psachin has quit IRC | 13:30 | |
| fungi | pabelanger: given the support timeframe and release frequency for debian is roughly comparable to ubuntu lts it would make sense to treat the two of them similarly. maybe we should consider ubuntu-lts and centos-latest too though? | 13:30 |
| *** gouthamr has joined #openstack-infra | 13:32 | |
| *** mriedem has joined #openstack-infra | 13:32 | |
| fungi | pabelanger: the reason that message from january just showed up is because i've (we've) been lax about checking the moderator queue and it was sent by a non-subscriber so only just got approved yesterday | 13:32 |
| pabelanger | fungi: I'm of the mind, I don't think ubuntu or centos has to change, since we ways care multiple releases of those DIBs, eg, trusty / xenial / bionic. But if we only plan on having a single debian stable online at a time, and not using stable branches (like we do for fedora), I could get behind debian-stable / fedora-latest | 13:32 |
| *** andreas_s has joined #openstack-infra | 13:32 | |
| pabelanger | fungi: Oh, that explains it | 13:33 |
| pabelanger | fungi: thanks | 13:33 |
| pabelanger | s/way care/ always carry/ | 13:34 |
| * pabelanger off to get coffee | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| *** dhajare has quit IRC | 13:37 | |
| *** andreas_s has quit IRC | 13:37 | |
| *** esberglu has joined #openstack-infra | 13:38 | |
| *** agopi has joined #openstack-infra | 13:38 | |
| *** olaph has joined #openstack-infra | 13:40 | |
| *** olaph1 has quit IRC | 13:40 | |
| *** gfidente has quit IRC | 13:41 | |
| *** andreas_s has joined #openstack-infra | 13:41 | |
| *** yamamoto has joined #openstack-infra | 13:43 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** gfidente has joined #openstack-infra | 13:45 | |
| *** caphrim007 has quit IRC | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** andreas_s has quit IRC | 13:46 | |
| *** olaph1 has joined #openstack-infra | 13:47 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:47 | |
| *** olaph has quit IRC | 13:47 | |
| *** sree has quit IRC | 13:48 | |
| *** sree has joined #openstack-infra | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** yamamoto has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| pabelanger | odyssey4me: if AJaeger doesn't get them shortly, I can +A in a bit | 13:52 |
| odyssey4me | tyvm | 13:52 |
| *** sree has quit IRC | 13:53 | |
| *** apetrich has quit IRC | 13:54 | |
| dmsimard | odyssey4me: are you working on a openstack-infra-like setup ? I see mentions of windmill | 13:55 |
| *** udesale__ has joined #openstack-infra | 13:56 | |
| prometheanfire | pabelanger: :D | 13:56 |
| AJaeger | pabelanger: go for it ;) | 13:57 |
| *** olaph1 has left #openstack-infra | 13:57 | |
| pabelanger | prometheanfire: if you want to propose the patch to project-config, I have some time today to monitor | 13:57 |
| *** dbecker has quit IRC | 13:57 | |
| prometheanfire | pabelanger: sure, I'll see if I can find it | 13:58 |
| prometheanfire | for now, meeting | 13:58 |
| *** annp has joined #openstack-infra | 13:59 | |
| *** eernst has joined #openstack-infra | 14:00 | |
| odyssey4me | dmsimard yep, windmill is pabelanger's project to do that - and we're using the ansible role components to do our own nodepool implementation | 14:01 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova-multiattach libvirt perm denied bug 1763382 https://review.openstack.org/560914 | 14:02 |
| openstack | bug 1763382 in OpenStack Compute (nova) ""libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" in nova-multiattach job since we started using the Queens UCA" [High,Confirmed] https://launchpad.net/bugs/1763382 | 14:02 |
| *** hemna_ has joined #openstack-infra | 14:02 | |
| *** zhipeng_ has joined #openstack-infra | 14:02 | |
| dmsimard | odyssey4me: you might be interested in http://lists.openstack.org/pipermail/openstack-infra/2018-April/005897.html | 14:02 |
| *** iyamahat has joined #openstack-infra | 14:02 | |
| odyssey4me | dmsimard great - hopefully we can work together on the roles! :) | 14:03 |
| *** jpich has quit IRC | 14:03 | |
| *** jaosorior has joined #openstack-infra | 14:04 | |
| *** r-daneel has quit IRC | 14:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 14:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 14:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 14:05 |
| Shrews | odyssey4me: your "own nodepool implementation"? Is that a replacement for zuulv3's nodepool? | 14:05 |
| *** VW has joined #openstack-infra | 14:06 | |
| fungi | my guess would have been nodepool without zuul | 14:07 |
| *** jpich has joined #openstack-infra | 14:07 | |
| *** lpetrut has quit IRC | 14:08 | |
| *** mugsie has quit IRC | 14:08 | |
| *** lpetrut has joined #openstack-infra | 14:08 | |
| dmsimard | I understood it as their deployment of nodepool (vs re-implementing a different nodepool-like software) | 14:09 |
| *** armaan has joined #openstack-infra | 14:12 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:12 | |
| *** e0ne has joined #openstack-infra | 14:12 | |
| *** xarses has quit IRC | 14:12 | |
| *** hongbin has joined #openstack-infra | 14:13 | |
| *** andreykurilin_ has quit IRC | 14:14 | |
| *** rpioso|afk is now known as rpioso | 14:15 | |
| AJaeger | mordred: did you see http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html ? That relates to the tox-siblings work. Btw. did you write up that email explaing that work already? | 14:15 |
| mriedem | so uh, i'm seeing something really weird with zuulv3 not picking up a change | 14:16 |
| mriedem | https://review.openstack.org/#/c/554317/3/playbooks/legacy/nova-multiattach/run.yaml removes the ENABLE_UBUNTU_CLOUD_ARCHIVE=False line right? | 14:17 |
| mriedem | but it's still here http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/ara-report/result/ab8397a6-593e-43f5-a593-c2633a0e40de/ | 14:17 |
| mriedem | causing that job to fail | 14:17 |
| mriedem | this was totally fine until yesterday i guess | 14:17 |
| mriedem | i know there was a code deploy last night | 14:17 |
| mriedem | i assume zuulv3 bugs are tracked in storyboard... | 14:18 |
| fungi | mriedem: yeah, the sb project link is in the zuul docs, but https://storyboard.openstack.org/#!/project/679 | 14:20 |
| mriedem | https://storyboard.openstack.org/#!/story/2001839 | 14:20 |
| pabelanger | mriedem: seems to be picking up playbooks on stable/queens http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/job-output.txt.gz#_2018-04-11_18_03_39_919225 | 14:21 |
| mriedem | huh, that's less than good | 14:22 |
| pabelanger | I'm not too sure why that is ATM | 14:23 |
| pabelanger | http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/zuul-info/inventory.yaml shows _inheritance_path | 14:23 |
| *** dpawlik has quit IRC | 14:24 | |
| *** owalsh_afk is now known as owalsh | 14:28 | |
| clarkb | kashyap: for the pcid feature in particult right? | 14:29 |
| kashyap | clarkb: Yes. | 14:29 |
| kashyap | clarkb: Note that: Queens and Pike will allow *only* PCID to be specified. | 14:29 |
| kashyap | I.e. no other CPU flags are allowed. | 14:29 |
| kashyap | We did that so as to keep the backport to stable branches as reasonable as possible. | 14:30 |
| kashyap | For Rocky, in an upcoming patch, I intend to lift that restriction, and allow free-form CPU flags to be specified -- making way for other features. | 14:30 |
| clarkb | kashyap: cool and it has a verbose release note so hopefully deployers take advantage of that | 14:32 |
| kashyap | clarkb: Yeah, spent a month or so refining that & getting the details right (/me is a slow writer) | 14:33 |
| *** jbadiapa has quit IRC | 14:33 | |
| * kashyap gets back to doing some "unfun" activity that he _must_ finish by tomorrow | 14:33 | |
| *** r-daneel has joined #openstack-infra | 14:35 | |
| fungi | mriedem: pabelanger: http://zuul.openstack.org/ does show jobs running for 554317,4 (queued as of 7 minutes ago) | 14:35 |
| mriedem | i just rebased | 14:36 |
| mriedem | to put it on top of the patch to make that job non-voting since it's blocking nova (and cinder) now | 14:36 |
| fungi | and the zuul scheduler log claims there were jobs running for 554317,3 triggered by the recheck prior to that at 13:34 | 14:36 |
| *** dklyle has joined #openstack-infra | 14:36 | |
| *** udesale__ has quit IRC | 14:36 | |
| mriedem | yeah i rechecked it b/c i saw the n-cpu startup timeout in the devstack logs, | 14:36 |
| *** udesale has joined #openstack-infra | 14:37 | |
| mriedem | but that was before i realized that n-cpu was fialing to start b/c of failing to connect to libvirt, which is because of the localrc being wrong | 14:37 |
| *** derekh_ has quit IRC | 14:37 | |
| mriedem | which is due to the zuul thing | 14:37 |
| odyssey4me | Shrews no, no - sorry - we're deploying nodepool and using it with jenkins... we've published the jenkins plugin for nodepool too: https://github.com/jenkinsci/nodepool-agents-plugin | 14:37 |
| fungi | ahh, when you said zuul wasn't picking up the change, you mean it seemed to be checking out the wrong source code, not that it wasn't triggering any jobs to run? | 14:37 |
| clarkb | is this the multiattach job? | 14:38 |
| clarkb | if so frickler and I debugged it last night ish at different times and the problem is I think devstack stackrc special cases multiattach and does the wrong thing | 14:38 |
| clarkb | I'm not entirely sure its a zuul problem in that case | 14:38 |
| fungi | yeah, nova-multiattach | 14:40 |
| fungi | mriedem: so based on that, the regression you've seen may be due to a recent change in devstack behavior around it | 14:40 |
| fungi | rather than due to the zuul scheduler restart yesterday | 14:41 |
| clarkb | mriedem: frickler identified https://review.openstack.org/#/c/554314/ as the likely cause | 14:41 |
| clarkb | mriedem: which is your change :P | 14:41 |
| mriedem | yes, i realize that change, | 14:41 |
| mriedem | which is why we have https://review.openstack.org/#/c/554317/ | 14:41 |
| *** markvoelker has quit IRC | 14:42 | |
| mriedem | which was queued up to merge right after the devstack change so we wouldn't blow up the gate, | 14:42 |
| mriedem | however, | 14:42 |
| mriedem | as noted in the SB bug, | 14:42 |
| mriedem | zuul isn't picking up that playbook change on master | 14:42 |
| mriedem | https://review.openstack.org/#/c/554317/4/playbooks/legacy/nova-multiattach/run.yaml | 14:42 |
| *** markvoelker has joined #openstack-infra | 14:42 | |
| mriedem | http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/ara-report/result/ab8397a6-593e-43f5-a593-c2633a0e40de/ | 14:42 |
| mriedem | those are not equal | 14:42 |
| clarkb | ok now I'm up to speed on what the zuul issue is | 14:42 |
| *** beagles is now known as beagles_brb | 14:45 | |
| *** jcoufal has quit IRC | 14:45 | |
| *** markvoelker has quit IRC | 14:47 | |
| clarkb | mriedem: pabelanger somewhat guessing here but is https://git.openstack.org/cgit/openstack/nova/tree/.zuul.yaml#n99 possibly at fault? | 14:49 |
| clarkb | that shouldn't be necessary since zuul will only run the job on the branches you tell it to run it via the check and gate stuff at the bottom of the file | 14:49 |
| *** AR2018 has joined #openstack-infra | 14:49 | |
| mriedem | hmm, i can't remember why that is in there since the job def is per branch | 14:51 |
| *** rajinir has joined #openstack-infra | 14:51 | |
| mriedem | maybe just an old habit when i defined the job in queens | 14:51 |
| mriedem | i can remove it and see what happens | 14:52 |
| clarkb | specifically I'm wondering if that directive on the queens branch is allowing the queens config to match for master | 14:53 |
| clarkb | mriedem: so I think you may need to remove it from queens | 14:53 |
| clarkb | corvus: ^ | 14:53 |
| mriedem | can i remove it from both at the same time? | 14:53 |
| clarkb | no because they are on separate branches | 14:53 |
| mriedem | like, (1) make the job non-voting, (2) remove branch specifier on master (3) backport to queens (4) make the job voting again | 14:53 |
| *** jcoufal has joined #openstack-infra | 14:54 | |
| clarkb | mriedem: I'd do it the other way around fix queens then fix master | 14:54 |
| clarkb | I don't think a backport is necessary if queens itself is broken | 14:54 |
| *** mugsie has joined #openstack-infra | 14:54 | |
| *** mugsie has quit IRC | 14:54 | |
| *** mugsie has joined #openstack-infra | 14:54 | |
| mriedem | if the job is non-voting does it matter? | 14:54 |
| clarkb | (you should be able to confirm if this fixes it by using depends on) | 14:54 |
| clarkb | mriedem: no, its just less work to make two changes than 4 | 14:54 |
| mriedem | (1) is in the gate, | 14:54 |
| mriedem | (4) is https://review.openstack.org/#/c/554317/ | 14:54 |
| clarkb | (also this is still a bit of a theory at this point so getting it tested would be a good first step) | 14:55 |
| mriedem | i'll push some patches | 14:55 |
| *** VW_ has joined #openstack-infra | 14:55 | |
| *** VW_ has quit IRC | 14:55 | |
| *** gouthamr has quit IRC | 14:56 | |
| *** VW_ has joined #openstack-infra | 14:56 | |
| *** VW has quit IRC | 14:58 | |
| *** hamzy has quit IRC | 14:58 | |
| *** ralonsoh__ has quit IRC | 14:59 | |
| *** VW has joined #openstack-infra | 14:59 | |
| *** VW_ has quit IRC | 15:00 | |
| *** freerunner has quit IRC | 15:00 | |
| *** SergeyLukjanov has quit IRC | 15:00 | |
| *** ruhe has quit IRC | 15:00 | |
| *** NikitaKonovalov has quit IRC | 15:00 | |
| *** DinaBelova has quit IRC | 15:00 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Restrict legacy-tempest-dsvm-neutron-src job https://review.openstack.org/559839 | 15:01 |
| mriedem | clarkb: https://review.openstack.org/#/q/topic:bug/1763382+(status:open+OR+status:merged) | 15:01 |
| dmellado | clarkb: fungi I was wondering about how does the upstream gate installs the pip limits. I've got a patch capping kubernetes version with upper reqs on the kuryr tempest plugin which seems to be completely ignored | 15:02 |
| dmellado | https://review.openstack.org/#/c/560885/ | 15:02 |
| *** r-daneel has quit IRC | 15:02 | |
| *** gouthamr has joined #openstack-infra | 15:02 | |
| *** ramishra has quit IRC | 15:02 | |
| clarkb | dmellado: kubernetes isn't a python dependency so pip and constraints and requirements shouldn't affect it right? | 15:03 |
| dmellado | clarkb: I'm referring to python-kubernetes client | 15:03 |
| clarkb | ah | 15:03 |
| dmellado | i.e. kubenetes within pip and so | 15:03 |
| dmellado | https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L350 | 15:04 |
| *** mugsie has quit IRC | 15:04 | |
| *** efoley has joined #openstack-infra | 15:04 | |
| clarkb | dmellado: do you have a link to a failing job run we can look at logs for? | 15:05 |
| clarkb | mriedem: reading https://docs.openstack.org/infra/zuul/user/config.html#attr-job.branches it says "Otherwise, all of the job variants which match that branch (and any other selection criteria) are used when freezing the job." which I think explains this behavior we see | 15:06 |
| *** mugsie has joined #openstack-infra | 15:06 | |
| *** mugsie has quit IRC | 15:06 | |
| *** mugsie has joined #openstack-infra | 15:06 | |
| clarkb | however reading that I'm not entirely sure how a user can expect reliable results in that case | 15:06 |
| clarkb | corvus: ^ | 15:06 |
| fungi | dmellado: looks like 560885 is adding constraints to the tox jobs for the openstack/kuryr-tempest-plugin but is that actually what you're seeing break, or are you wanting it constrained in devstack jobs? | 15:06 |
| *** annp has quit IRC | 15:07 | |
| dmellado | clarkb: fungi yep, but when that one ends | 15:07 |
| dmellado | you'll see that it fails | 15:08 |
| *** mugsie has quit IRC | 15:08 | |
| dmellado | and the failure is related to a change in python kubernetes | 15:08 |
| *** mugsie has joined #openstack-infra | 15:08 | |
| *** mugsie has quit IRC | 15:08 | |
| *** mugsie has joined #openstack-infra | 15:08 | |
| dmellado | which shows only in 6 | 15:08 |
| dmellado | and not in 5 | 15:08 |
| *** germs has quit IRC | 15:08 | |
| *** apetrich has joined #openstack-infra | 15:09 | |
| dmellado | https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#delete_namespaced_service | 15:09 |
| *** germs has joined #openstack-infra | 15:09 | |
| *** germs has quit IRC | 15:09 | |
| *** germs has joined #openstack-infra | 15:09 | |
| dmellado | this asks for name, namespace and body | 15:09 |
| corvus | clarkb: catching up | 15:09 |
| dmellado | and that body requirement has been only added in 5 | 15:09 |
| dmellado | in 6, I mean | 15:09 |
| dmellado | so I don't really know why this 6 is installed | 15:09 |
| dmellado | locally, I end up with kubernetes 6 on system-site-packages and 5 within the tox env | 15:09 |
| clarkb | dmellado: do you have logs we can look at? | 15:10 |
| clarkb | dmellado: this is all very specific to how the job is running and what actually runs | 15:10 |
| dmellado | clarkb: it'll be shown soon on that patch | 15:10 |
| dmellado | it is but I wonder if this wouldn't affect all the tempest plugins | 15:10 |
| dmellado | as openstack is BC but might be happening as well | 15:10 |
| dmellado | I'm also deploying on a server from my side and will be back with more info, just wanted to know if it ringed a bell | 15:11 |
| dmellado | thanks in any case! | 15:11 |
| *** HeOS has joined #openstack-infra | 15:12 | |
| dmellado | fungi: basically is as you say, I'd need that constrained for the zuul dsvm job | 15:12 |
| clarkb | devstack constraints all installs if you do them via the pip_install function | 15:13 |
| clarkb | we'll have to read the logs to see what is going on | 15:14 |
| *** belliott has quit IRC | 15:14 | |
| dmellado | ack, I'll ping you back when they're around | 15:14 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for swap volume active block job failure bug 1763410 https://review.openstack.org/560938 | 15:14 |
| dmellado | thanks! | 15:14 |
| openstack | bug 1763410 in OpenStack Compute (nova) "swap volume intermittently fails with "libvirtError: block copy still active: domain has active block job"" [Medium,Confirmed] https://launchpad.net/bugs/1763410 | 15:14 |
| corvus | clarkb: iiuc, you, pabelanger, and mriedem identified that the variant of multiattach on queens was applying to master changes because it had an explicit branch matcher which matches master. that's going to be removed, so the job will rely on implicit branch matchers everywhere. is that correct? | 15:15 |
| *** DinaBelova has joined #openstack-infra | 15:15 | |
| *** jaosorior has quit IRC | 15:15 | |
| *** eernst has quit IRC | 15:15 | |
| *** NikitaKonovalov has joined #openstack-infra | 15:16 | |
| *** eernst has joined #openstack-infra | 15:16 | |
| *** ruhe has joined #openstack-infra | 15:16 | |
| *** freerunner has joined #openstack-infra | 15:16 | |
| pabelanger | corvus: I think so, mriedem proposed an update to 554317 removing the branch matcher | 15:17 |
| *** SergeyLukjanov has joined #openstack-infra | 15:17 | |
| *** beagles_brb is now known as beagles | 15:19 | |
| *** hamzy has joined #openstack-infra | 15:21 | |
| clarkb | corvus: yup | 15:22 |
| corvus | clarkb: your last comment to me indicated some confusion "I'm not entirely sure how a user can expect reliable results in that case" -- can you elaborate? | 15:22 |
| clarkb | corvus: ya, the change was to the master branch but the job graph freezing order seemed more arbitrary (it applied queens then master not master then queens) | 15:23 |
| clarkb | corvus: a bit confused at how that behavior would be used in a useful fashion in a way that users would find determinstic | 15:23 |
| corvus | clarkb: actually it did master then queens | 15:24 |
| corvus | clarkb: it is deterministic. it's master, followed by all other branches in alpha order | 15:24 |
| corvus | if you look at http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/zuul-info/inventory.yaml | 15:25 |
| clarkb | http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/zuul-info/inventory.yaml | 15:25 |
| clarkb | corvus: ya it did master after queens there. if I read this right | 15:25 |
| *** sree has joined #openstack-infra | 15:25 | |
| clarkb | (the first master listed was the check: jobs: this-job config based on line numebrs) | 15:25 |
| corvus | yeah, there are 3 'nova-multiattach' jobs at the end. the job on master, the job on queens, then the project-pipeline job on master | 15:25 |
| clarkb | and by first I mean last because reverse order | 15:25 |
| clarkb | https://git.openstack.org/cgit/openstack/nova/tree/.zuul.yaml?id=c810b9432839ae0fcb361cdbc80158a2a00b5454#n114 | 15:26 |
| corvus | now i'm confused... i don't think anything is in reverse order | 15:26 |
| clarkb | corvus: first applied is not first in the file | 15:27 |
| clarkb | or maybe that is my confusion | 15:27 |
| corvus | clarkb: the project-pipeline variant is always the last thing applied | 15:27 |
| clarkb | gotcha | 15:27 |
| corvus | all the variants before it are file-ordered | 15:28 |
| clarkb | with that in mind. This behavior will likely very rarely if at all do the right thing for openstack projects with master and stable branches | 15:28 |
| clarkb | because it will always apply the stable branch configs after master configs | 15:28 |
| corvus | so basically, when you invoke a job in a project pipeline, all the matching variants get applied in branch-file order, then the project-pipeline gets to modify the thing right before it's run | 15:29 |
| corvus | clarkb: on the contrary: exactly the right thing will happen, as long as you don't have any explicit branch matchers | 15:29 |
| clarkb | thats what I mean | 15:29 |
| clarkb | https://docs.openstack.org/infra/zuul/user/config.html#attr-job.branches specifically will do the wrong thing for openstack usually | 15:30 |
| corvus | i think the issue here is that auto-migrated jobs got copy-pastad from project-config into a project repo, then backported | 15:30 |
| *** e0ne has quit IRC | 15:30 | |
| *** sree has quit IRC | 15:30 | |
| clarkb | (whcih is fine we can say don't use jobs.branches) | 15:30 |
| corvus | clarkb: no. not to put too fine a point on it, but instead of saying "branches: ^(?!stable/(newton|ocata|pike)).*$" say "" | 15:30 |
| clarkb | don't use explicit jobs.branches values | 15:31 |
| clarkb | is that more accurate? | 15:31 |
| *** markvoelker has joined #openstack-infra | 15:31 | |
| corvus | yeah, they should be avoided for in-repo configs unless you're doing something really complicated. | 15:31 |
| clarkb | or reading the docs maybe "rely on the implied jobs.branches value, don't try to modify it in the job directly" | 15:31 |
| *** andreas_s has joined #openstack-infra | 15:32 | |
| *** gyee has joined #openstack-infra | 15:32 | |
| corvus | i very much doubt this was a concious choice though. we probably should have had something in the infra-manual v3 migration docs that says "hey, if you copy a legacy job to your repo, remove the branches attr and backport it to the branches you want it to run on" | 15:32 |
| *** felipemonteiro__ has joined #openstack-infra | 15:33 | |
| clarkb | ya it was likely cargo culted | 15:33 |
| dulek | clarkb: Regarding dmellado investigation: https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L645-L654 | 15:34 |
| dulek | clarkb: That's not constrainted install. | 15:34 |
| pabelanger | ianw: just noticed, we are setting block-device-mbr in http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh#n118 but not nodepool.yaml. Why would that be? | 15:34 |
| dulek | clarkb: I believe this is the issue. Now I don't know if it's a bug or there's a reason for that. | 15:35 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 15:35 |
| dmellado | dulek: also check that that | 15:35 |
| dmellado | tox -evenv-tempest | 15:35 |
| dmellado | in the end calls this | 15:35 |
| dmellado | https://github.com/openstack/tempest/blob/master/tox.ini#L6-L13 | 15:35 |
| clarkb | dulek: ya that is likely the issue, a few lines above that it does use constraints to isntall tempest though | 15:35 |
| dmellado | which should be using constraints | 15:35 |
| dmellado | clarkb: I assume you're calling that code from the tempest cloned dir, isn't it | 15:36 |
| dmellado | ? | 15:36 |
| dulek | dmellado: It's only for venv creation, I think. | 15:36 |
| dmellado | dulek: but it goes inside tempest dir | 15:36 |
| clarkb | ya that directive in tox.ini will only manage the creation | 15:36 |
| clarkb | then it modifies it in the devstack script that dulek pointed out | 15:36 |
| dmellado | but https://github.com/openstack/tempest/blob/master/tox.ini#L145-L150 | 15:37 |
| *** felipemonteiro_ has quit IRC | 15:37 | |
| dmellado | takes from tempesvenv | 15:37 |
| dmellado | which has the constraints | 15:37 |
| pabelanger | clarkb: corvus: on the subject deterministic above "it's master, followed by all other branches in alpha order" how does that come into play where projects have develop branch, then master, then other branches. I would assume develop would fall into alpha order of other branches? | 15:37 |
| dmellado | oh, nvm | 15:37 |
| dmellado | it just installs | 15:38 |
| dmellado | yep | 15:38 |
| dulek | dmellado: As clarkb said - it will use them only for venv creation not for pip executions. | 15:38 |
| dulek | dmellado: Yup. | 15:38 |
| clarkb | pabelanger: yes that was my understanding. Basically master is the only special branch there | 15:38 |
| dmellado | so we'd need to put a constraints there as well | 15:38 |
| clarkb | pabelanger: everything else is alpha order | 15:38 |
| dulek | dmellado: So I guess we have a bug found. | 15:38 |
| *** caphrim007 has joined #openstack-infra | 15:38 | |
| dmellado | it has been just 'working' because the openstack api is bc | 15:38 |
| dmellado | and kubernetees isn't | 15:38 |
| pabelanger | clarkb: okay, good to know | 15:39 |
| corvus | pabelanger, clarkb: yeah. we can/should alter the configloader to make the 'default branch' the special one, but right now it's hardcoded. | 15:39 |
| *** zhipeng_ has quit IRC | 15:39 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 15:39 |
| pabelanger | +1 | 15:39 |
| *** markvoelker_ has joined #openstack-infra | 15:40 | |
| clarkb | corvus: it may be worth a note in teh zuul jobs.branches doc that the typical user should rely on the implied behavior and not try to override it. Overriding is an advanced feature and there be dragons sort of deal? | 15:41 |
| *** caphrim007 has quit IRC | 15:42 | |
| *** caphrim007 has joined #openstack-infra | 15:42 | |
| *** caphrim007 has quit IRC | 15:42 | |
| *** caphrim007 has joined #openstack-infra | 15:43 | |
| *** markvoelker has quit IRC | 15:44 | |
| *** dhajare has joined #openstack-infra | 15:45 | |
| *** zhipeng_ has joined #openstack-infra | 15:49 | |
| corvus | clarkb: hrm. i'm not sure this is the bast case for that since, after all, this job wasn't exactly created as the result of reading that documentation. | 15:49 |
| *** edmondsw has joined #openstack-infra | 15:50 | |
| corvus | clarkb: i think the doc update should be to the infra v3 migration guide, if we think that's still relevant. | 15:50 |
| clarkb | at this point it probably isn't, maybe just a note to the -dev list pointing out if you are in this situation a cleanup may be desireable | 15:50 |
| corvus | ++ | 15:50 |
| clarkb | if mriedem confirms this is the fix (I expect it will be at this point) I can send that message | 15:50 |
| corvus | thx | 15:50 |
| *** iyamahat has quit IRC | 15:52 | |
| *** yamahata has quit IRC | 15:53 | |
| *** edmondsw has quit IRC | 15:53 | |
| pabelanger | we are getting a lot of failures for nodepool in rackspace currently | 15:55 |
| pabelanger | looking to see why | 15:55 |
| *** sree has joined #openstack-infra | 15:55 | |
| pabelanger | seems to be SSH timeouts, starting at 04:45 UTC | 15:56 |
| pabelanger | check to see if we uploaded new images at that time | 15:56 |
| *** edmondsw has joined #openstack-infra | 15:58 | |
| *** neiloy_ has joined #openstack-infra | 15:58 | |
| *** l4yerffej has quit IRC | 15:59 | |
| pabelanger | time line seem to line up, 00:11:22:20 ago new xenial images came online | 15:59 |
| clarkb | changes to config drive maybe breaking glean? | 16:00 |
| *** felipemonteiro__ has quit IRC | 16:00 | |
| clarkb | might need to manually boot one and use the web console log thing | 16:00 |
| pabelanger | we did release a new DIB yesterday | 16:00 |
| *** trown is now known as trown|lunch | 16:01 | |
| pabelanger | let me boot one and look | 16:01 |
| pabelanger | then we can pause images on rax and roll back | 16:02 |
| *** zhipeng_ has quit IRC | 16:02 | |
| pabelanger | it seems to be only ubuntu that is affected | 16:03 |
| *** efried has quit IRC | 16:03 | |
| *** efried has joined #openstack-infra | 16:04 | |
| clarkb | we changed dns stuff but that shouldn't break ssh I don't think | 16:04 |
| clarkb | unless its breaking on reverse dns lookups? | 16:04 |
| clarkb | pabelanger: you might try manual ssh too to test ^ with a longer timeout | 16:04 |
| pabelanger | oh, maybe | 16:05 |
| pabelanger | booting node now | 16:06 |
| *** l4yerffej has joined #openstack-infra | 16:07 | |
| *** jpich has quit IRC | 16:07 | |
| *** iyamahat has joined #openstack-infra | 16:09 | |
| *** panda|rover is now known as panda|rover|off | 16:10 | |
| *** dbecker has joined #openstack-infra | 16:10 | |
| *** lucasagomes is now known as lucas-afk | 16:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove zuul-cloner command https://review.openstack.org/560958 | 16:11 |
| *** armaan has quit IRC | 16:15 | |
| *** armaan has joined #openstack-infra | 16:15 | |
| pabelanger | okay, console pulled up now | 16:16 |
| pabelanger | not much in way of errors | 16:17 |
| pabelanger | time to enter resume mode | 16:17 |
| pabelanger | rescue* | 16:18 |
| clarkb | pabelanger: have you tried ssh directly yourself? | 16:18 |
| clarkb | if it is a dns issue it might not show in console log | 16:18 |
| pabelanger | clarkb: yah, connection timeout | 16:18 |
| *** dbecker has quit IRC | 16:19 | |
| pabelanger | will check to see if glean ran first | 16:19 |
| *** florianf has quit IRC | 16:20 | |
| *** dbecker has joined #openstack-infra | 16:20 | |
| pabelanger | image was built using 2.13.0 | 16:21 |
| *** shardy has quit IRC | 16:21 | |
| pabelanger | previous images has same version of glean too | 16:22 |
| *** neiloy_ has quit IRC | 16:24 | |
| fungi | can you ping them but you get a socket timeout connecting on 22/tcp? | 16:25 |
| *** e0ne has joined #openstack-infra | 16:25 | |
| *** yamahata has joined #openstack-infra | 16:27 | |
| pabelanger | yah, no ping. Looks like we didn't setup networking | 16:28 |
| pabelanger | having some issues getting rescue mode working, trying again | 16:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Publish docs in release pipeline https://review.openstack.org/560964 | 16:29 |
| *** udesale has quit IRC | 16:32 | |
| *** garyk has quit IRC | 16:36 | |
| pabelanger | clarkb: fungi : http://paste.openstack.org/show/719071/ | 16:38 |
| pabelanger | that is on inap | 16:38 |
| pabelanger | so looks like an issue with resolv.conf | 16:38 |
| *** gouthamr has quit IRC | 16:38 | |
| pabelanger | check other providers | 16:38 |
| pabelanger | still haven't been able to log into rax | 16:38 |
| pabelanger | ovh seems okay | 16:39 |
| pabelanger | let me check another inap | 16:41 |
| mriedem | clarkb: fungi: we might want to promote this https://review.openstack.org/#/c/560909/ | 16:41 |
| mriedem | the gate looks to be pretty f'ed up otherwise, since at least all nova/cinder patches are going to get kicked out | 16:42 |
| clarkb | pabelanger: that shouldn't prevent network interfaces from coming up though right? so possibly two issues | 16:42 |
| pabelanger | clarkb: fungi: okay, inap seems okay. that must have been ajob that updated it | 16:42 |
| clarkb | mriedem: corvus can probably confirm but are we sure that makes the job non voting? | 16:42 |
| clarkb | mriedem: since it is applying the queens config too. | 16:43 |
| mriedem | oh and tempest | 16:43 |
| mriedem | ugh | 16:43 |
| * clarkb looks at status page to see if it is voting | 16:43 | |
| clarkb | ok it appears to not be voting on the status page so it worked I guess | 16:43 |
| mriedem | whew | 16:43 |
| clarkb | I'll enqueue it to the gate and promote it now | 16:44 |
| mriedem | thanks | 16:44 |
| corvus | that will make it non-voting on master. queens will be voting. | 16:45 |
| pabelanger | clarkb: yah, I'm having a hard time getting rescue mode working here, still no network. I am unsure if rackspace is using our image to mount the filesystem or one of their own images | 16:45 |
| clarkb | pabelanger: rescue mode should be their own image to boot then they attach our image next to it | 16:45 |
| clarkb | s/next// | 16:45 |
| pabelanger | clarkb: yah, that is what I am expecting too | 16:46 |
| clarkb | mriedem: ok should be done now | 16:46 |
| mriedem | thanks again | 16:46 |
| *** yamamoto has quit IRC | 16:47 | |
| pabelanger | clarkb: I'm going to open a ticket work rax and see what is going on. In the mean time, I think we can pause DIBs for ubuntu-xenial and roll back nodepool to the previous images | 16:47 |
| *** andreas_s has quit IRC | 16:49 | |
| *** jamesmcarthur has quit IRC | 16:49 | |
| *** jamesmcarthur has joined #openstack-infra | 16:50 | |
| *** udesale has joined #openstack-infra | 16:51 | |
| *** dpawlik has joined #openstack-infra | 16:52 | |
| *** jbadiapa has joined #openstack-infra | 16:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Pause ubuntu-xenial DIBs https://review.openstack.org/560971 | 16:53 |
| clarkb | we might be able to boot the old image then apply new glean. Thought I guess you said glean version didn't change | 16:53 |
| pabelanger | infra-root: if you don't mind reviewing^, I'd like to revert to the previous version of ubuntu-xenial for nodepool while we debug why latest version is failing on rackspace only | 16:54 |
| pabelanger | clarkb: yah, same glean version | 16:54 |
| *** jamesmcarthur has quit IRC | 16:55 | |
| pabelanger | clarkb: I'm going to rebuild a local image with devuser to use console login | 16:56 |
| pabelanger | while I wait for reply from rax support | 16:56 |
| bgmccollum_ | pabelanger rax uses a coreos rescue image | 16:56 |
| *** VW has quit IRC | 16:56 | |
| *** dpawlik has quit IRC | 16:56 | |
| *** dtruong_ has quit IRC | 16:56 | |
| clarkb | pabelanger: that seems like a reasonable next step (using dev user with passwd) | 16:56 |
| bgmccollum_ | at least last time i used it | 16:56 |
| *** udesale has quit IRC | 16:56 | |
| pabelanger | bgmccollum_: good to know thanks, any idea why network would not come up on it? | 16:56 |
| *** dtruong has joined #openstack-infra | 16:57 | |
| bgmccollum_ | all instance of the same image fail to come up? | 16:57 |
| *** udesale has joined #openstack-infra | 16:58 | |
| *** udesale has quit IRC | 16:58 | |
| * fungi needs to disappear for an appointment but should return by ~19:00z | 16:58 | |
| *** yamamoto has joined #openstack-infra | 16:58 | |
| pabelanger | bgmccollum_: in this case yes, ubuntu-xenial, it is our image we've upload and suspect we broke something. But using the rescue mode option doesn't seem to enable networking (if using coreos on that image) | 16:58 |
| bgmccollum_ | pabelanger im running a rescue test right now...let me see what happens. | 16:59 |
| *** iyamahat_ has joined #openstack-infra | 17:02 | |
| *** iyamahat has quit IRC | 17:02 | |
| *** yamamoto has quit IRC | 17:03 | |
| *** dbecker has quit IRC | 17:03 | |
| bgmccollum_ | pabelanger it appears rescue uses the same image as the instance was originally booted from. when i said coreos, i was thinking about rescue mode of onmetal servers. so if the original image was broken from the beginning from a network perspective, the it will be broken is rescue mode too im guessing. | 17:03 |
| *** neiloy_ has joined #openstack-infra | 17:03 | |
| bgmccollum_ | pabelanger have you tried using the console + rescue mode to poke around? | 17:03 |
| clarkb | bgmccollum_: I think that was going to be the next step (build an image with a local user that don't depend on boot time config to set up) | 17:06 |
| *** electrofelix has quit IRC | 17:06 | |
| *** dklyle has quit IRC | 17:07 | |
| *** belliott has joined #openstack-infra | 17:07 | |
| *** VW has joined #openstack-infra | 17:08 | |
| *** bgmccollum_ is now known as bgmccollum | 17:08 | |
| *** janki has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova-multiattach libvirt perm denied bug 1763382 https://review.openstack.org/560914 | 17:10 |
| AJaeger | mnaser: could you review https://review.openstack.org/548404 again, please? | 17:10 |
| openstack | bug 1763382 in OpenStack Compute (nova) ""libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" in nova-multiattach job since we started using the Queens UCA" [High,In progress] https://launchpad.net/bugs/1763382 - Assigned to Matt Riedemann (mriedem) | 17:10 |
| *** hashar is now known as hasharAway | 17:11 | |
| *** jamesmcarthur has joined #openstack-infra | 17:11 | |
| *** betherly_ has joined #openstack-infra | 17:12 | |
| *** trown|lunch is now known as trown | 17:12 | |
| *** russellb has quit IRC | 17:13 | |
| *** tjgresha has joined #openstack-infra | 17:13 | |
| *** gouthamr has joined #openstack-infra | 17:14 | |
| *** mugsie has quit IRC | 17:14 | |
| *** mugsie has joined #openstack-infra | 17:15 | |
| *** mugsie has quit IRC | 17:15 | |
| *** mugsie has joined #openstack-infra | 17:15 | |
| *** jamesmcarthur has quit IRC | 17:15 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add build-sphinx check/gate jobs https://review.openstack.org/560090 | 17:17 |
| *** iranzo has quit IRC | 17:17 | |
| *** jpena is now known as jpena|off | 17:17 | |
| pabelanger | bgmccollum: clarkb: sorry, somebody at door. Yah, building new image with console access now | 17:18 |
| *** efoley has quit IRC | 17:19 | |
| *** neiloy_ has quit IRC | 17:20 | |
| *** hamzy_ has joined #openstack-infra | 17:20 | |
| *** belliott has quit IRC | 17:21 | |
| *** russellb has joined #openstack-infra | 17:21 | |
| *** hamzy has quit IRC | 17:22 | |
| openstackgerrit | Merged openstack-infra/zuul master: zuul-changes: update for the new api url https://review.openstack.org/557656 | 17:23 |
| *** zoli is now known as zoli|gone | 17:26 | |
| *** zoli|gone is now known as zoli | 17:26 | |
| *** pblaho has quit IRC | 17:26 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/irc-meetings master: Change Horizon meeting time https://review.openstack.org/560979 | 17:29 |
| *** iranzo has joined #openstack-infra | 17:29 | |
| *** andreas_s has joined #openstack-infra | 17:29 | |
| *** jamesmcarthur has joined #openstack-infra | 17:31 | |
| *** corvus is now known as jeblair | 17:31 | |
| *** jeblair is now known as corvus | 17:31 | |
| *** kjackal_ has quit IRC | 17:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 17:34 |
| *** andreas_s has quit IRC | 17:34 | |
| *** Hal has quit IRC | 17:35 | |
| *** jamesmcarthur has quit IRC | 17:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Parent tox-py35-on-zuul on tox-py35 https://review.openstack.org/560982 | 17:38 |
| pabelanger | I've promoted 560971 into gate to help speed up the pause of ubuntu-xenial dibs, since we are down capacity for nodepool | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove zuul-tox-py35 https://review.openstack.org/560983 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 17:39 |
| *** iranzo has quit IRC | 17:40 | |
| *** AR2018 has quit IRC | 17:40 | |
| *** wolverineav has joined #openstack-infra | 17:44 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add Gerrit docs to Zuul From Scratch https://review.openstack.org/558600 | 17:44 |
| *** lpetrut has quit IRC | 17:44 | |
| *** amoralej is now known as amoralej|off | 17:45 | |
| *** sambetts is now known as sambetts|afk | 17:48 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add static driver doc to Zuul From Scratch https://review.openstack.org/558802 | 17:50 |
| *** eharney_ has joined #openstack-infra | 17:51 | |
| *** gfidente has quit IRC | 17:51 | |
| *** eharney has quit IRC | 17:51 | |
| *** eharney_ is now known as eharney | 17:51 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add example nodepool-launcher systemd service file https://review.openstack.org/560993 | 17:53 |
| *** tesseract has quit IRC | 17:55 | |
| *** tjgresha has quit IRC | 17:56 | |
| *** dsariel has quit IRC | 17:58 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 17:58 |
| *** jtomasek has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create ansible-role-whisper repository https://review.openstack.org/560061 | 18:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Create ansible-role-carbon repository https://review.openstack.org/560058 | 18:02 |
| openstackgerrit | Merged openstack-infra/nodepool master: Handle ZK session loss during node launch https://review.openstack.org/552538 | 18:03 |
| *** aojea has quit IRC | 18:03 | |
| *** aojea has joined #openstack-infra | 18:04 | |
| *** yamamoto has joined #openstack-infra | 18:04 | |
| *** dklyle has joined #openstack-infra | 18:05 | |
| *** wolverineav has quit IRC | 18:06 | |
| *** wolverin_ has joined #openstack-infra | 18:06 | |
| *** eernst has quit IRC | 18:08 | |
| *** eernst has joined #openstack-infra | 18:09 | |
| *** pcaruana has quit IRC | 18:09 | |
| *** wolverineav has joined #openstack-infra | 18:10 | |
| *** wolverin_ has quit IRC | 18:10 | |
| openstackgerrit | Merged openstack-infra/zuul master: gerrit: recognize project-created event https://review.openstack.org/560274 | 18:11 |
| *** felipemonteiro has joined #openstack-infra | 18:17 | |
| *** sree has quit IRC | 18:18 | |
| *** aojea has quit IRC | 18:18 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:18 | |
| *** sree has joined #openstack-infra | 18:18 | |
| *** jtomasek has joined #openstack-infra | 18:19 | |
| *** openstackgerrit has quit IRC | 18:19 | |
| *** jamesmcarthur has joined #openstack-infra | 18:20 | |
| *** felipemonteiro has quit IRC | 18:22 | |
| AJaeger | mordred: did you see http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html ? doesn't that relate to the tox-siblings work? Btw. did you write up that email explaing that work already? | 18:23 |
| mordred | AJaeger: I don't thnk it relates to siblings - but no, I still haven't written that up :( | 18:24 |
| *** openstackgerrit has joined #openstack-infra | 18:26 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: allow building non-gentoo images on gentoo hosts https://review.openstack.org/561002 | 18:26 |
| AJaeger | mordred: So, only relates to the build-sphinx work - where extras are ignored? | 18:26 |
| openstackgerrit | Tim Rozet proposed openstack/diskimage-builder master: Removes systemd condition for dhcp-interface https://review.openstack.org/561003 | 18:27 |
| *** links has joined #openstack-infra | 18:31 | |
| *** sree has quit IRC | 18:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add example nodepool-launcher systemd service file https://review.openstack.org/560993 | 18:33 |
| *** felipemonteiro_ has quit IRC | 18:37 | |
| *** lpetrut has joined #openstack-infra | 18:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 18:38 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented scheduler PID file default https://review.openstack.org/561010 | 18:39 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 18:43 |
| *** belliott has joined #openstack-infra | 18:44 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 18:44 |
| mordred | AJaeger: oh - it's possible I need to read more of the thread | 18:45 |
| *** pcichy has joined #openstack-infra | 18:49 | |
| openstackgerrit | Merged openstack-infra/zuul master: configloader: add variant-description https://review.openstack.org/549748 | 18:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Pause ubuntu-xenial DIBs https://review.openstack.org/560971 | 18:53 |
| *** r-daneel has joined #openstack-infra | 18:54 | |
| corvus | pabelanger: ^ | 18:54 |
| *** sree has joined #openstack-infra | 18:54 | |
| mordred | AJaeger: thanks - responded | 18:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 18:55 |
| pabelanger | corvus: ack | 18:55 |
| *** jamesmcarthur has quit IRC | 18:55 | |
| *** jamesmcarthur has joined #openstack-infra | 18:56 | |
| *** jamesmcarthur has quit IRC | 18:56 | |
| AJaeger | mordred: thansk | 18:57 |
| *** sree has quit IRC | 18:59 | |
| *** sree has joined #openstack-infra | 19:00 | |
| *** jamesmcarthur has joined #openstack-infra | 19:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 19:01 |
| *** trown has quit IRC | 19:03 | |
| *** eharney has quit IRC | 19:03 | |
| *** salv-orlando has quit IRC | 19:04 | |
| *** salv-orlando has joined #openstack-infra | 19:04 | |
| dmsimard | pabelanger: comments on https://review.openstack.org/#/c/560897/ | 19:04 |
| *** sree has quit IRC | 19:05 | |
| *** trown has joined #openstack-infra | 19:06 | |
| openstackgerrit | Merged openstack-infra/nodepool master: launcher: handle ZK session loss during handler poll https://review.openstack.org/556335 | 19:07 |
| dmsimard | There's no meeting between 4PM and 5PM (EST) according to calendar, I'll merge changes which require a meetbot restart | 19:08 |
| pabelanger | dmsimard: replied | 19:08 |
| *** salv-orlando has quit IRC | 19:09 | |
| pabelanger | okay, I've deleted ubuntu-xenial-0000001395 rackspace should be coming back online in a moment | 19:09 |
| pabelanger | I've also gotten console access to the DIB I manually built, for some reason glean isn't getting run. Trying to see what changed | 19:10 |
| dmsimard | EmilienM, mwhahaha: do you know the answer to clarkb's question in https://review.openstack.org/#/c/558934/ ? | 19:12 |
| EmilienM | dmsimard: it's good to go | 19:12 |
| dmsimard | nothing special to do ? | 19:12 |
| EmilienM | dmsimard: we probably need to import bugs from launchpad | 19:13 |
| *** felipemonteiro has joined #openstack-infra | 19:13 | |
| EmilienM | dmsimard: bugs with "validation" tag | 19:13 |
| *** sshnaidm|mtg is now known as sshnaidm|afk | 19:13 | |
| EmilienM | diablo_rojo: ^ right? | 19:13 |
| dmsimard | EmilienM: ok there's no tripleo-validations launchpad ? | 19:13 |
| EmilienM | there is tripleo launchpad with validation tag | 19:13 |
| EmilienM | that's what we'll import | 19:13 |
| *** felipemonteiro_ has joined #openstack-infra | 19:14 | |
| EmilienM | https://bugs.launchpad.net/tripleo/+bugs?field.tag=validations | 19:14 |
| EmilienM | validations actually | 19:14 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add debian-stable nodeset https://review.openstack.org/560897 | 19:15 |
| *** sree has joined #openstack-infra | 19:15 | |
| pabelanger | okay, rackspace working again with old images. Back to debugging why latests are failing to start glean | 19:16 |
| *** trown has quit IRC | 19:17 | |
| *** felipemonteiro has quit IRC | 19:17 | |
| *** trown has joined #openstack-infra | 19:17 | |
| *** agopi has quit IRC | 19:18 | |
| prometheanfire | pabelanger: not going to unpause until I fix something gentoo infra side that keeps on breaking (image doesn't match checksum, file transfer truncated) | 19:20 |
| * prometheanfire is annoyed it's still an issue and is going to fix it himself | 19:20 | |
| *** sree has quit IRC | 19:20 | |
| *** gouthamr has quit IRC | 19:20 | |
| *** sree has joined #openstack-infra | 19:21 | |
| dmsimard | AJaeger: I would tend to -1 https://review.openstack.org/#/c/537876/3 | 19:21 |
| dmsimard | AJaeger: if they're going to modify legacy jobs, they should carry it in tree | 19:21 |
| diablo_rojo | EmilienM, correct. Going to migrate bugs with that tag next. | 19:24 |
| EmilienM | cool | 19:24 |
| dmsimard | diablo_rojo: ok so there's no issue in approving https://review.openstack.org/#/c/558934/ ? | 19:24 |
| *** sree has quit IRC | 19:25 | |
| *** krenczewski has joined #openstack-infra | 19:25 | |
| *** sree has joined #openstack-infra | 19:26 | |
| pabelanger | okay, booting working glean xenial to see what is the difference | 19:27 |
| pabelanger | wonder if a package update has changed something to do with systemd ordering | 19:28 |
| *** krenczew1 has quit IRC | 19:28 | |
| *** belliott has quit IRC | 19:28 | |
| *** dbecker has joined #openstack-infra | 19:30 | |
| *** gouthamr has joined #openstack-infra | 19:30 | |
| *** andreas_s has joined #openstack-infra | 19:30 | |
| *** sree has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove zuul-cloner command https://review.openstack.org/560958 | 19:31 |
| openstackgerrit | Merged openstack-infra/zuul master: Publish docs in release pipeline https://review.openstack.org/560964 | 19:31 |
| fungi | infra-root: clarkb has noticed that etherpad tagged another release (1.6.5) with some more bug fixes including a security fix (related to untrusted plugins so not a bug deal) but i'm going to restart the etherpad-lite service on etherpad-dev.openstack.org just so we're running and testing that release on it now | 19:32 |
| fungi | #status log restarted etherpad-lite service on review-dev.openstack.org to pick up commits related to latest 1.6.5 tag | 19:33 |
| openstackstatus | fungi: finished logging | 19:33 |
| fungi | #status log restarted etherpad-lite service on etherpad-dev.openstack.org (NOT review-dev!) to pick up commits related to latest 1.6.5 tag | 19:33 |
| openstackstatus | fungi: finished logging | 19:34 |
| *** wolverin_ has joined #openstack-infra | 19:34 | |
| *** wolverineav has quit IRC | 19:34 | |
| corvus | fungi: it may not be a big deal, but it's definitely a bug deal! | 19:34 |
| clarkb | https://etherpad-dev.openstack.org/p/foo-test is quick test | 19:35 |
| diablo_rojo | dmsimard, no issue at all :) | 19:35 |
| *** dhajare has quit IRC | 19:35 | |
| fungi | corvus: ahh, i'll just add that one to my typo karma debt | 19:35 |
| *** andreas_s has quit IRC | 19:35 | |
| * fungi sighs at his typos upon typos | 19:35 | |
| AJaeger | dmsimard: yes, it's time to push back for those... | 19:36 |
| clarkb | tyops you mean | 19:36 |
| corvus | clarkb: tyops.io is available if you want to register it and start promoting the latest foo-ops craze. | 19:37 |
| *** sree has joined #openstack-infra | 19:38 | |
| fungi | thank-you, ops! | 19:38 |
| mordred | corvus, clarkb: you could make it have an ansible fork where all keywords are replaced with typo alternates | 19:39 |
| clarkb | fungi: thanks driven operations is not a bad idea | 19:39 |
| fungi | sounds like a new term for how we operate | 19:40 |
| fungi | please fix this kthxbye! | 19:40 |
| *** agopi has joined #openstack-infra | 19:41 | |
| *** sree has quit IRC | 19:43 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Make the yaml parser aware of '!encrypted/' tags https://review.openstack.org/559963 | 19:44 |
| *** jamesmcarthur has quit IRC | 19:45 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 19:45 |
| *** jamesmcarthur has joined #openstack-infra | 19:46 | |
| *** links has quit IRC | 19:46 | |
| *** linkmark has quit IRC | 19:49 | |
| *** gouthamr has quit IRC | 19:50 | |
| *** gouthamr has joined #openstack-infra | 19:51 | |
| clarkb | fungi: we ready to start an etherpad.o.o upgrade? I'm guessing the first step is to fix the homedir, then update nodejs, then update the etherpad ref to 1.6.5 and restart the service? | 19:56 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: role: Inject public keys in case of failure https://review.openstack.org/535803 | 19:57 |
| fungi | clarkb: sounds right | 19:57 |
| fungi | infra-root: ^ i guess anyone who wants to do some last-minute testing on etherpad-dev.o.o should do so now | 19:58 |
| *** pcichy has quit IRC | 19:58 | |
| fungi | i'll also do a db dump right before the upgrade | 19:58 |
| *** jamesmcarthur has quit IRC | 19:59 | |
| *** jcoufal_ has joined #openstack-infra | 19:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Track tripleo-validations tasks on StoryBoard https://review.openstack.org/558934 | 20:00 |
| *** iyamahat_ has quit IRC | 20:00 | |
| fungi | i'll stage this in two commits i guess, since we need to do the nodejs manual upgrade step between the nodejs_version change and the eplite_version change | 20:00 |
| *** iyamahat has joined #openstack-infra | 20:01 | |
| clarkb | fungi: sounds good | 20:02 |
| *** jcoufal has quit IRC | 20:02 | |
| *** felipemonteiro_ has quit IRC | 20:03 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:03 | |
| *** jcoufal_ has quit IRC | 20:04 | |
| *** snapiri has quit IRC | 20:04 | |
| *** oanson has quit IRC | 20:05 | |
| *** salv-orlando has joined #openstack-infra | 20:05 | |
| *** VW_ has joined #openstack-infra | 20:07 | |
| *** salv-orlando has quit IRC | 20:09 | |
| *** salv-orlando has joined #openstack-infra | 20:09 | |
| *** e0ne has quit IRC | 20:10 | |
| *** VW has quit IRC | 20:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Test base job secrets https://review.openstack.org/561030 | 20:11 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update to nodejs 6.x on etherpad.o.o https://review.openstack.org/561031 | 20:11 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update etherpad to 1.6.5 https://review.openstack.org/561032 | 20:11 |
| fungi | clarkb: ^ | 20:11 |
| *** snapiri has joined #openstack-infra | 20:11 | |
| *** VW_ has quit IRC | 20:11 | |
| *** VW has joined #openstack-infra | 20:12 | |
| *** oanson has joined #openstack-infra | 20:12 | |
| *** jcoufal has joined #openstack-infra | 20:12 | |
| fungi | we can approve the first one lazily since the manual step is non-impacting and i've wip'd the second change | 20:13 |
| fungi | pabelanger: dmsimard: ianw: ^ related to things we were talking about a couple days ago | 20:14 |
| clarkb | fungi: I've +2'd both | 20:14 |
| *** sree has joined #openstack-infra | 20:15 | |
| *** agopi has quit IRC | 20:15 | |
| fungi | #status log manually corrected the eplite homedir path in /etc/passwd on etherpad.o.o and created it on the filesystem with appropriate ownership following https://review.openstack.org/528625 | 20:19 |
| *** sree has quit IRC | 20:19 | |
| openstackstatus | fungi: finished logging | 20:19 |
| openstackgerrit | Merged openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 20:20 |
| openstackgerrit | Merged openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 20:20 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 20:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Reduce test concurrency by 1 cpu https://review.openstack.org/561037 | 20:21 |
| *** yamamoto has quit IRC | 20:21 | |
| *** yamamoto has joined #openstack-infra | 20:22 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: start converting oslo libraries to build docs with py3 https://review.openstack.org/561038 | 20:22 |
| *** kgiusti has left #openstack-infra | 20:23 | |
| *** markvoelker_ has quit IRC | 20:23 | |
| *** agopi has joined #openstack-infra | 20:25 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 20:27 |
| *** jamesmcarthur has joined #openstack-infra | 20:27 | |
| fungi | dmsimard: did you see the parent change (561031)? | 20:27 |
| fungi | it can merge immediately if you're good with it, and then i'll run the manual step it mentions on the server once it takes effect | 20:28 |
| fungi | which may be after i get back from grabbing an early dinner | 20:28 |
| dmsimard | fungi: got sidetracked in the middle of reviewing things | 20:31 |
| *** jamesmcarthur has quit IRC | 20:32 | |
| *** dsariel has joined #openstack-infra | 20:32 | |
| dmsimard | infra-root: looks like the the pipelines are recovering from a provider outage so the queues are a bit backed up: http://grafana.openstack.org/dashboard/db/nodepool | 20:34 |
| fungi | dmsimard: no worries. just thought you might have missed that change had an open parent | 20:34 |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| *** wolverin_ has quit IRC | 20:34 | |
| *** wolverineav has joined #openstack-infra | 20:34 | |
| dmsimard | looks like RAX | 20:35 |
| dhellmann | I would like to create a variation of the release notes job that uses python 3 instead of 2. Can someone give me a little guidance about how to do that? | 20:35 |
| dmsimard | http://grafana.openstack.org/dashboard/db/nodepool-rackspace | 20:35 |
| clarkb | dmsimard: yes, its a known issue. Xenial instances stopped booting there and pabelanger is investigating | 20:36 |
| clarkb | dhellmann: yes, let me find an example job that has a similar switch | 20:36 |
| dmsimard | looks like it's recovering | 20:36 |
| corvus | dmsimard: because pabelanger reverted to the previous image; we don't know what the problem is yet. | 20:37 |
| *** sree has joined #openstack-infra | 20:38 | |
| corvus | clarkb: zuul sphinx-build-docs is one | 20:38 |
| clarkb | corvus: yup just getting links for that one now | 20:39 |
| clarkb | dhellmann: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/zuul.yaml#n218 is the job | 20:39 |
| corvus | kk. didn't want to step on toes :) | 20:39 |
| *** wolverineav has quit IRC | 20:39 | |
| clarkb | now just looking for where that var is consumed | 20:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix typo in build-sphinx-docs docstring https://review.openstack.org/561040 | 20:41 |
| dhellmann | clarkb : I see that build-sphinx-docs and build-reno-releasenotes both use the ensure-sphinx role and it appears that is what consumes the variable | 20:41 |
| clarkb | dhellmann: yup that is the location | 20:41 |
| dhellmann | does that mean that I can just use the variable with build-reno-releasenotes? or do I need to do something to cause it to be passed into the job (other than setting it like I see in publish-openstack-sphinx-docs-python3) | 20:42 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/ensure-sphinx/tasks/main.yaml#n34 for anyone looking through irc logs | 20:42 |
| *** VW has quit IRC | 20:42 | |
| clarkb | dhellmann: I think you can just set it on the job | 20:42 |
| dhellmann | cool | 20:42 |
| clarkb | dhellmann: zuul does this in its .zuul.yaml even | 20:42 |
| clarkb | for the docs builds | 20:43 |
| *** sree has quit IRC | 20:43 | |
| *** mriedem is now known as mriedem_funtime | 20:43 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: document that the build-reno-releasenotes job accepts sphinx_python https://review.openstack.org/561041 | 20:43 |
| *** gouthamr_ has joined #openstack-infra | 20:43 | |
| * fungi heads out to grab some food, will manually upgrade nodejs on etherpad.o.o when he gets back if 561031 merges while he's gone | 20:44 | |
| corvus | clarkb, dhellmann: yep: http://git.zuul-ci.org/cgit/zuul/tree/.zuul.yaml#n57 | 20:44 |
| *** tosky___ has joined #openstack-infra | 20:44 | |
| *** tosky has quit IRC | 20:45 | |
| *** tosky___ is now known as tosky | 20:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove openstack-101 from accessbot https://review.openstack.org/559250 | 20:45 |
| *** wolverineav has joined #openstack-infra | 20:46 | |
| dhellmann | it still feels weird to me that we have project templates in openstack-zuul-jobs referring to jobs defined in project-config | 20:47 |
| *** salv-orlando has quit IRC | 20:47 | |
| ianw | pabelanger: do i understand correctly that glean not starting on xenial hosts, rax only? | 20:47 |
| *** wolverineav has quit IRC | 20:47 | |
| *** salv-orlando has joined #openstack-infra | 20:48 | |
| clarkb | dhellmann: iirc that is because the jobs themselves need to be in trusted config but to ease testing we do as much as possible in the untrusted config (ozj) | 20:48 |
| *** wolverineav has joined #openstack-infra | 20:48 | |
| *** hamzy_ has quit IRC | 20:49 | |
| clarkb | ianw: that is my understanding of the situation | 20:49 |
| clarkb | if I had to guess somethign related to config drive handling broke maybe? | 20:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: document that the build-reno-releasenotes job accepts sphinx_python https://review.openstack.org/561041 | 20:49 |
| clarkb | since everything else does dhcp | 20:50 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: add release-notes-jobs-python3 project template https://review.openstack.org/561044 | 20:50 |
| pabelanger | ianw: yah, I am trying to debug it now. I am thinking there is something stopping us from mounting config-drive for some reason | 20:50 |
| pabelanger | but once I get access and run glean manually it works | 20:50 |
| pabelanger | so, I am thinking a race some place | 20:50 |
| dhellmann | clarkb : ok. I think I just don't get the expected sequence of changes so I may just toss a bunch of patches up and get someone to help me work out the depends-on stuff :-) | 20:51 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove openstack-101 from status and meetbot https://review.openstack.org/559249 | 20:52 |
| dhellmann | clarkb : like does this patch to oslo.config do what I expect and test the new project template? https://review.openstack.org/561046 | 20:53 |
| clarkb | dhellmann: if you add a depends on and fix the syntax error in the parent, yes | 20:53 |
| dhellmann | oops | 20:53 |
| clarkb | dhellmann: I left comment for the syntax error | 20:54 |
| ianw | pabelanger: how bizarre, considering we install & boot as part of the nodepool-src tests ... only slight difference for ubuntu is https://review.openstack.org/#/c/504588/ that i can think of | 20:54 |
| clarkb | ianw: its the only cloud we rely on config drive to set up the interfaces with | 20:54 |
| clarkb | ianw: every other cloud dhcps | 20:55 |
| pabelanger | ianw: yah, I look at that but could see how it was related | 20:55 |
| clarkb | including devstack in nodepool-src | 20:55 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: add release-notes-jobs-python3 project template https://review.openstack.org/561044 | 20:56 |
| pabelanger | glean is trying to start, but then something is stopping it | 20:56 |
| pabelanger | I haven't figured out yet what | 20:56 |
| *** VW has joined #openstack-infra | 20:58 | |
| pabelanger | ianw: do you maybe want to try a revert of 504588 and upload image to rax to see if that is it? | 20:58 |
| pabelanger | I'm still trying to get systemd to print more debug logs | 20:58 |
| *** agopi has quit IRC | 20:58 | |
| ianw | pabelanger: re your question on mbr element, we have a hack to patch it in as the default if it's not seen https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/lib/common-functions#n242 | 20:59 |
| dhellmann | clarkb : it doesn't seem to be running the new jobs. I see build-openstack-sphinx-docs but not build-openstack-sphinx-docs-python3 | 20:59 |
| *** agopi has joined #openstack-infra | 20:59 | |
| dhellmann | oh, though I guess the job name won't be different, will it? | 20:59 |
| *** eharney has joined #openstack-infra | 21:00 | |
| clarkb | dhellmann: correct | 21:00 |
| clarkb | dhellmann: you'll want to check the version of python used in the logs | 21:00 |
| dhellmann | and it's not running the release notes job at all, which probably means there's an exclusion | 21:00 |
| ianw | pabelanger: ok, what's the minimal element list? ubuntu-minimal simple-init replicates it? | 21:00 |
| clarkb | dhellmann: probably need to update a release note, I debuged a similar issue with rosmaita yesterday and release notes do filter on files pretty aggresively | 21:01 |
| pabelanger | ianw: http://paste.openstack.org/show/719099/ is what I used | 21:03 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** VW_ has joined #openstack-infra | 21:04 | |
| *** rlandy is now known as rlandy|afk | 21:05 | |
| *** gouthamr has quit IRC | 21:06 | |
| *** slaweq has quit IRC | 21:07 | |
| *** jamesmcarthur has quit IRC | 21:08 | |
| *** slaweq has joined #openstack-infra | 21:08 | |
| *** VW has quit IRC | 21:08 | |
| tonyb | gema, pabelanger: There are small amount of hw avauilable but right now it isn't somethign we're workign on | 21:09 |
| *** VW_ has quit IRC | 21:09 | |
| dhellmann | clarkb : yep, that was it | 21:09 |
| johnsom | Are any projects using the non-legacy multinode parents? Our multi-node job is broken-ish and I am thinking about just reworking it to be "native" | 21:10 |
| johnsom | Looking for an example project I can look at | 21:10 |
| clarkb | johnsom: I'm not sure, you might be able to use codesearch.openstack.org to quickly check though | 21:11 |
| pabelanger | [ 3.933014] systemd[1]: system-glean.slice: Job system-glean.slice/start finished, result=canceled | 21:11 |
| pabelanger | that is what systemd debug log is saying | 21:11 |
| pabelanger | still don't know what canceled it | 21:11 |
| johnsom | Yeah, kind of looked there, searched for openstack-infra-multinode-integration-ubuntu-xenial and didn't get a hit outside of infra | 21:12 |
| *** jamesmcarthur has joined #openstack-infra | 21:12 | |
| clarkb | johnsom: andreaf would be the other person to check with | 21:12 |
| johnsom | Ok, thanks. Just looking for a pointer to "a good way to do it". | 21:12 |
| *** slaweq has quit IRC | 21:13 | |
| ianw | pabelanger: ok, building | 21:14 |
| *** gouthamr has joined #openstack-infra | 21:15 | |
| pabelanger | ianw: clarkb: i think if we can find the syntax to make sure the blkid for config-2 is online, that might be the trick | 21:15 |
| pabelanger | there is a way for systemd to query that, testing | 21:15 |
| ianw | src/core/job.h: JOB_CANCELED, /* Job canceled by a conflicting job installation or by explicit cancel request */ | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| pabelanger | if only it said why :) | 21:17 |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/python-storyboardclient master: remove sphinx-isms from README.rst and clean up docs https://review.openstack.org/545447 | 21:18 |
| pabelanger | I thought systemd-udev-settle.service might be the service but don't believe so | 21:19 |
| ianw | pabelanger: looking at the code more, https://www.freedesktop.org/software/systemd/man/systemctl.html#--job-mode= "isolate" and "flush" mode are a possibility | 21:21 |
| ianw | https://www.freedesktop.org/software/systemd/man/systemctl.html#isolate%20UNIT | 21:22 |
| ianw | Start the unit specified on the command line and its dependencies and stop all others, unless they have IgnoreOnIsolate=yes | 21:22 |
| pabelanger | systemd has a .mount target, I wonder if we should maybe think about moving /mnt/config to /etc/fstab over our glean.sh script | 21:23 |
| ianw | IgnoreOnIsolate= ... If true, this unit will not be stopped when isolating another unit. Defaults to false for service, target, socket, busname, timer, and path units, and true for slice, scope, device, swap, mount, and automount units. | 21:24 |
| *** Goneri has quit IRC | 21:24 | |
| *** rfolco is now known as rfolco|off | 21:25 | |
| *** dpawlik has joined #openstack-infra | 21:25 | |
| ianw | i wonder if the run-level is changing and that's cancelling system-glean.slice | 21:25 |
| *** openstackstatus has quit IRC | 21:27 | |
| *** openstack has joined #openstack-infra | 21:28 | |
| *** ChanServ sets mode: +o openstack | 21:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update to nodejs 6.x on etherpad.o.o https://review.openstack.org/561031 | 21:30 |
| *** dpawlik has quit IRC | 21:30 | |
| *** sree has joined #openstack-infra | 21:32 | |
| *** jamesmcarthur has quit IRC | 21:32 | |
| *** hasharAway has quit IRC | 21:33 | |
| *** lpetrut has quit IRC | 21:35 | |
| *** sambetts|afk has quit IRC | 21:36 | |
| *** sree has quit IRC | 21:36 | |
| *** sambetts_ has joined #openstack-infra | 21:38 | |
| *** sree has joined #openstack-infra | 21:38 | |
| pabelanger | ianw: clarkb: I'm stepping away for a bit to grab food and family time. But, haven't made any major progress yet :( | 21:38 |
| *** sree has quit IRC | 21:43 | |
| dhellmann | clarkb : it looks like it's working: http://logs.openstack.org/46/561046/3/check/build-openstack-releasenotes/b9f1962/job-output.txt.gz#_2018-04-12_21_31_08_626428 | 21:47 |
| clarkb | dhellmann: ya that looks like it did the right thing | 21:47 |
| dhellmann | clarkb : thanks for your help on putting that together | 21:47 |
| clarkb | dhellmann: I've +2'd the two related infra changes | 21:48 |
| dhellmann | awesome, thanks for that, too | 21:48 |
| dhellmann | I'll work with bnemec and dims on the oslo.config changes | 21:48 |
| *** agopi has quit IRC | 21:49 | |
| ianw | clarkb / pabelanger: hmm, i can't log into the account we use to run ci hosts on rax "Account temporarily locked: Too many failed login attempts." | 21:49 |
| efried | How does a guy cycle meetbot in a channel? | 21:49 |
| ianw | and i can not for the life of me remember my personal login | 21:50 |
| melwitt | yes, meetbot in #openstack-meeting won't respond to #endmeeting. need help | 21:50 |
| clarkb | ianw: the nodepool account is doing that? | 21:50 |
| clarkb | melwitt: efried anyone can end a meeting one hour after it started | 21:51 |
| efried | clarkb: meetbot isn't responding tho | 21:51 |
| melwitt | yeah, it didn't respond to the last topic change and then now it won't end the meeting that I started | 21:51 |
| clarkb | however the problem is meetbot got restarted so isn't running any meetings | 21:51 |
| efried | ah, so the meeting died a silent death at some point in there. | 21:52 |
| melwitt | restarted in the middle of our meeting? so we won't get the log or minutes | 21:52 |
| efried | That means the chair has to write 'em up by hand. Bummer. | 21:52 |
| efried | :P | 21:52 |
| *** jtomasek has quit IRC | 21:52 | |
| melwitt | heh. well, I can copy-paste the log | 21:52 |
| *** gouthamr has quit IRC | 21:54 | |
| clarkb | melwitt: efried I think https://review.openstack.org/559249 was the culprit | 21:55 |
| *** sree has joined #openstack-infra | 21:55 | |
| ianw | dmsimard: what do you remember about clearing out old images on rax? there's *a lot* there now | 21:56 |
| efried | clarkb: Thanks for the info. | 21:56 |
| clarkb | dmsimard: ^ as a general heads up we can't update the meetbot config without restarting it so we'll want to approve those changes for when no meetings are in progress | 21:56 |
| dmsimard | clarkb: there were no meetings on the calendar when I approved the change | 21:57 |
| dmsimard | And it merged before the next meeting :/ | 21:57 |
| clarkb | dmsimard: ya but has to wait for puppet pulse | 21:57 |
| clarkb | so you need like ~45 minutes + time to merge | 21:57 |
| *** esberglu has quit IRC | 21:58 | |
| dmsimard | Ah, apologies melwitt, my fault :( | 21:58 |
| dmsimard | ianw: I never got around to cleaning it up | 21:58 |
| dmsimard | ianw: I believe nodepool loses track of the images | 21:58 |
| dmsimard | Because rax overwrites the properties that nodepool sets on the images with their own | 21:59 |
| ianw | dmsimard: ok, that explains why they're not cleaned up as a first thing :) | 21:59 |
| melwitt | dmsimard: that's okay. glad to understand what happened, it was mysterious to us | 21:59 |
| *** jcoufal has quit IRC | 21:59 | |
| clarkb | ianw: dmsimard in that case lets try to get the cleanup done then we can monitor it going forward | 22:00 |
| *** sree has quit IRC | 22:00 | |
| *** bobh has quit IRC | 22:00 | |
| dmsimard | I'm preparing dinner but I can follow up later | 22:00 |
| *** eharney has quit IRC | 22:03 | |
| *** felipemonteiro__ has joined #openstack-infra | 22:08 | |
| *** jamesmcarthur has joined #openstack-infra | 22:10 | |
| *** jmorgan1 has joined #openstack-infra | 22:12 | |
| *** felipemonteiro_ has quit IRC | 22:12 | |
| ianw | clarkb / dmsimard : ok, i've got the active images via nodepool image-list | grep rax | awk '{print $12}' | tr '\n' '|' | 22:12 |
| *** tpsilva has quit IRC | 22:15 | |
| *** felipemonteiro__ has quit IRC | 22:15 | |
| ianw | clarkb / dmsimard: which leaves me with -> http://paste.openstack.org/show/719108/ ... any other thoughts before i start removing them? | 22:15 |
| *** felipemonteiro__ has joined #openstack-infra | 22:16 | |
| *** lbragstad has quit IRC | 22:16 | |
| clarkb | ianw: nope except that your paste got cut off so don't use that data as source :) | 22:16 |
| clarkb | ianw: oh the other thing is the images are actually in swift | 22:16 |
| clarkb | mordred: ^ do we have to also delete the image in swift too or will that happen automagically? | 22:17 |
| *** salv-orlando has quit IRC | 22:18 | |
| ianw | clarkb: oh, maybe "cat file | xclip" doesn't work that great | 22:18 |
| clarkb | ianw: I think it may have to do with size limits on the pastes you can post to our server | 22:18 |
| *** salv-orlando has joined #openstack-infra | 22:19 | |
| andreaf | johnsom yeah multinode works on master, the devstack base integration job can work multinode as well - just use the multinode node-set | 22:19 |
| andreaf | johnsom the tempest base job devstack-tempest is not multinode capable yet, I'm going through the devstack backport needed for that | 22:20 |
| johnsom | andreaf Yeah, experimenting with that now. Oh, then I suspect my experiment is going to fail... | 22:20 |
| andreaf | johnsom tosky and dmellado are doing multinode jobs for sahara and kuryr as well | 22:21 |
| *** AR2018 has joined #openstack-infra | 22:21 | |
| ianw | alright, in other news i have an image i can maybe boot to see this glean issue there now | 22:22 |
| johnsom | andreaf Is there a patch I can track on the devstack-tempest work? | 22:22 |
| andreaf | johnsom https://review.openstack.org/#/c/545724/ | 22:22 |
| andreaf | stable jobs fail there because orchestrate-devstack in on master only | 22:23 |
| johnsom | andreaf Awesome, thank you! | 22:23 |
| *** salv-orlando has quit IRC | 22:23 | |
| andreaf | https://review.openstack.org/#/q/topic:multinode_zuulv3+(status:open+OR+status:merged) | 22:23 |
| clarkb | ianw: I think maybe what we want to do is delete one image then see if we can confirm the swift image was deleted too | 22:23 |
| clarkb | and if so then do the rest and if not do swift and glance deletes together? | 22:24 |
| mordred | clarkb: no | 22:24 |
| mordred | wait | 22:24 |
| mordred | that was too short of an answer | 22:24 |
| *** harlowja has joined #openstack-infra | 22:24 | |
| mordred | clarkb: deleting the image in glance directly will not cause the objects uploaded to swift as part of uploading the image to glance to be deleted | 22:24 |
| mordred | clarkb: however- shade should have logic to delete the swift objects | 22:25 |
| ianw | hmm, we have ~60TB of images | 22:25 |
| clarkb | yay for leaking | 22:26 |
| ianw | mordred: hmm, i'm "image delete" ing centos now | 22:26 |
| clarkb | I expect that current shade + current nodepool + zk does track this far better than we have in the past | 22:26 |
| clarkb | its now just a matter of cleaning up the old stuff and checking again in a week or month for new leaks | 22:26 |
| mordred | clark, ianw: cloud.delete_image('foo', delete_objects=True) | 22:27 |
| mordred | will delete the image and the associated objects from swift | 22:27 |
| pabelanger | 60TB, ouch | 22:28 |
| mordred | ianw: I recommend making a 3 line shade script instead of using osc for image deletion tasks - however, if you don't want to do that, there sohuld be a metadata key on the image: owner_specified.shade.object | 22:29 |
| mordred | the value of that is the name of the container/object to delete in swift | 22:29 |
| ianw | mordred: i should have waited a bit, as i've now deleted the centos images with osc :) | 22:30 |
| *** rcernin has joined #openstack-infra | 22:30 | |
| mordred | sorry bout that :) | 22:30 |
| ianw | we've got two buckets, images and image_segments | 22:30 |
| ianw | images_segments | 22:30 |
| mordred | yah. image_segments is a behind the scenes things because the images are Large Objects in swift | 22:31 |
| *** rpioso is now known as rpioso|afk | 22:31 | |
| clarkb | we can likely clean up the swift stuff independently once we are done | 22:31 |
| clarkb | because I'm sure we've leaked there too | 22:31 |
| mordred | so the objects in the images/ container are manifest objects that refer to a set of segments in images_segments | 22:31 |
| mordred | we create the Large Objects in such a way that deleting the item in images/ should delete the corresponding segments | 22:31 |
| mordred | fwiw - none of the data in images/ or images_segments is precious - we could delete *ALL* of it if we wantedto and start fresh - it's only useful during the period of time that glance is importing the image | 22:32 |
| mordred | once the image is in glance, the swift objects are no longer used/needed | 22:32 |
| * mordred checked with rosmaita on that at the sydney summit | 22:32 | |
| clarkb | mordred: cool | 22:32 |
| ianw | ooohhhh, well that make it easier | 22:32 |
| clarkb | so lets delete the images first then delete the swift objects and be done | 22:33 |
| ianw | let me do it via shade for the rest | 22:33 |
| mordred | ianw, clarkb: we actually should not be leaking objects in swift at this point | 22:34 |
| clarkb | mordred: ya that was my reading of the situation when dmsimard looked semi recently | 22:34 |
| mordred | I just checked and shade is doing a delete_object once the image import is successful | 22:34 |
| clarkb | mordred: the combo of improving shade to handle this case and nodepool having a more reliable state tracking mechanism (zk) has lead to far better tracking of this stuff | 22:34 |
| rosmaita | mordred: you said "once the image is in glance, the swift objects are no longer used/needed" ... oh, you mean the object that glance doesn't own that you are importing into glance | 22:35 |
| rosmaita | ok, that's right, that isn't needed | 22:35 |
| rosmaita | i thought you were going to delete the segments that glance owned, that would be bad | 22:35 |
| *** Sukhdev has joined #openstack-infra | 22:35 | |
| ianw | rosmaita: we wouldn't be able to do that via the api? | 22:36 |
| rosmaita | no | 22:36 |
| rosmaita | so as long as you use the api you are fine | 22:36 |
| rosmaita | problems arise when people try to take shortcuts | 22:37 |
| *** dsariel has quit IRC | 22:37 | |
| *** eernst has quit IRC | 22:37 | |
| *** eernst has joined #openstack-infra | 22:38 | |
| mordred | rosmaita: no shortcuts here - only long cuts! | 22:38 |
| rosmaita | works for me! | 22:38 |
| fungi | shade is a vast collection of long, deep cuts | 22:38 |
| *** linkmark has joined #openstack-infra | 22:39 | |
| ianw | oh FFS!!!!! the host i'm doing all this on has disappeared | 22:40 |
| mordred | ianw: oh no! | 22:40 |
| ianw | why do people use clouds!!!!!! | 22:40 |
| mordred | ianw: they're terrible! | 22:40 |
| ianw | dmsimard: has rdocloud just experienced some sort of ... issue? | 22:41 |
| mordred | also - in a great example of redundancy, the power here just went out, but I'm still fine because now I'm tethered to my cell phone on my laptop | 22:41 |
| fungi | #status log nodejs has been manually nudged on etherpad.o.o to upgrade to 6.x packages now that https://review.openstack.org/561031 is in place | 22:42 |
| openstackstatus | fungi: finished logging | 22:42 |
| mordred | wow. unplanned outage: | 22:43 |
| mordred | Outage Begin Time 5:12 pm Thu, Apr, 12th | 22:44 |
| clarkb | mordred: is rdocloud hosted in your closet? | 22:44 |
| mordred | Estimated Restoration Time 7:00 pm Thu, Apr, 12th | 22:44 |
| mordred | Customers Out 49 | 22:44 |
| mordred | clarkb: maybe? | 22:44 |
| *** wolverineav has quit IRC | 22:44 | |
| clarkb | does that line up with your power outage? | 22:44 |
| *** wolverineav has joined #openstack-infra | 22:44 | |
| mordred | I mean - that's my power outage info | 22:45 |
| clarkb | oh | 22:45 |
| clarkb | for some reason I read that as rdooutage stuff | 22:45 |
| mordred | clarkb: that would have been more useful for me to paste | 22:45 |
| mordred | but woul dhave been less me talking about myself | 22:45 |
| *** dhill_ has quit IRC | 22:45 | |
| ianw | i'm going to do the school run, and hopefully when i come back everything will be back to working | 22:46 |
| ianw | pabelanger: i should then be able to boot a hoepfully affected image | 22:47 |
| clarkb | I need to step away for a bit too. Need something to drink | 22:47 |
| clarkb | maybe good too | 22:47 |
| pabelanger | ianw: ack | 22:47 |
| pabelanger | also, it is actually nice here today, 18C. that I am finally hacking from my front porch! | 22:48 |
| ianw | it's literally the hottest april ever recorded here and like 35C | 22:49 |
| clarkb | rainy here pretty typical | 22:49 |
| anteaya | ianw wow, still snowing here | 22:49 |
| *** sree has joined #openstack-infra | 22:50 | |
| anteaya | snow should be long gone | 22:50 |
| mattoliverau | +1 to the hottest April. I'm just glad I live on a beach ;) | 22:50 |
| mattoliverau | (and work inside) | 22:50 |
| anteaya | beach sounds nice | 22:52 |
| fungi | i probably shouldn't say sunny and 20c here at the beach | 22:52 |
| fungi | except i just did. sorry! | 22:52 |
| anteaya | ha ha ha | 22:54 |
| anteaya | that too sounds nice | 22:54 |
| anteaya | feels christmassy here | 22:54 |
| *** sree has quit IRC | 22:54 | |
| fungi | the breeze has stiffened a might | 22:55 |
| fungi | but that's typical for here | 22:55 |
| pabelanger | yah, there still was a lot of snow in Cochrane this week | 22:55 |
| pabelanger | and -30C nights :) | 22:55 |
| fungi | sustained 20-30kph winds here are fairly normal. makes it seem cooler than it is | 22:56 |
| pabelanger | ianw: I think bionic is also having an issue on rax, but I didn't pause and roll back that image. Since not may jobs should be using it ATM | 22:58 |
| *** hongbin has quit IRC | 22:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update etherpad to 1.6.5 https://review.openstack.org/561032 | 23:00 |
| anteaya | pabelanger, you were in Cochrane? | 23:00 |
| anteaya | fungi, sailing weather then? | 23:00 |
| fungi | anteaya: if i had a boat (or a sailboard, or a kiteboardm or...) then yes, absolutely | 23:01 |
| pabelanger | anteaya: yes, for about 5 days this past week. Back home now | 23:01 |
| anteaya | are you likely to aquire one of those items? | 23:01 |
| anteaya | pabelanger, hope you took your long johns | 23:01 |
| pabelanger | ++ | 23:02 |
| anteaya | glad common sense is still a thing | 23:02 |
| fungi | anteaya: maybe eventually, but next up is a couple of sea kayaks | 23:02 |
| anteaya | ohhhh, sea kayaks sounds wonderful | 23:03 |
| *** andreas_s has joined #openstack-infra | 23:03 | |
| *** caphrim007_ has joined #openstack-infra | 23:04 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 23:05 |
| anteaya | time to go load the outdoor furnace | 23:07 |
| *** caphrim007 has quit IRC | 23:08 | |
| *** caphrim007_ has quit IRC | 23:08 | |
| *** andreas_s has quit IRC | 23:09 | |
| *** vaidy has quit IRC | 23:14 | |
| *** isviridov_away has quit IRC | 23:14 | |
| clarkb | anteaya: is canada secretly running outdoor funrances to quicken the pace of global warming so taht your beachfront property values go up? | 23:16 |
| clarkb | or soon to be beachfront I should say | 23:16 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added CRON task to process eventbrite redeem promo codes https://review.openstack.org/561072 | 23:17 |
| *** eernst has quit IRC | 23:18 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added CRON task to process eventbrite redeem promo codes https://review.openstack.org/561072 | 23:19 |
| *** mriedem_funtime is now known as mriedem | 23:20 | |
| *** gouthamr has joined #openstack-infra | 23:21 | |
| mordred | pabelanger, ianw: I'm right between you at 27C - but that's not notable for april here | 23:24 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova xenapi unit test bug 1763535 https://review.openstack.org/561073 | 23:25 |
| openstack | bug 1763535 in OpenStack Compute (nova) "nova.tests.unit.virt.xenapi.test_vm_utils.StreamDiskTestCase.test_non_ami intermittently fails" [High,Confirmed] https://launchpad.net/bugs/1763535 | 23:25 |
| johnsom | Do I need to do something in my job definition to get bindep.txt picked up? This patch: https://review.openstack.org/#/c/559416 has debbootstrap in the bindep.txt but it's not getting installed: http://logs.openstack.org/16/559416/3/check/octavia-v1-dsvm-scenario/4b17b11/job-output.txt.gz#_2018-04-12_23_10_21_714351 | 23:28 |
| clarkb | johnsom: devstack doesn't use bindep | 23:28 |
| clarkb | there is a note on a wall somewhere to eventually make that transition but until that time devstack uses its own system of listing system deps | 23:28 |
| johnsom | That would explain my last hour and a half. hmm, ok, I will dig in devstack. Thanks clarkb | 23:30 |
| ianw | johnsom: yeah files/<apt|rpm> etc in devstack | 23:31 |
| johnsom | Yep, found it. Trying to move us over to ubuntu-minimal | 23:31 |
| anteaya | clarkb, well I'm not secretly running one in order to not freeze in my house | 23:33 |
| anteaya | I prefer to burn wood as opposed to natural gas | 23:33 |
| anteaya | now we can have an argument over which burns cleaner | 23:33 |
| mordred | anteaya: I hear you have plenty of wood up there | 23:33 |
| anteaya | we as a people do yes, and me as one human does as well | 23:34 |
| anteaya | now I'm all for reducing green house gas emmissions | 23:34 |
| clarkb | anteaya: ah so outdoor furnace heats the indoors? | 23:34 |
| anteaya | clarkb, yes, it has a a water bladder around the firebox and a line of water hoses into the house | 23:34 |
| anteaya | outdoor furnace is not designed to heat the outdoors no, though it does do some of that when I have the door open to put in more wood | 23:35 |
| mriedem | clarkb: you were right about the branch specifier in the stable/queens branch for the multiattach job: https://review.openstack.org/#/c/554317/6 | 23:35 |
| mriedem | that's passing now | 23:35 |
| clarkb | mriedem: cool. I'll send mail in the morning to the dev list warnings others of that particular behavior | 23:36 |
| anteaya | personally I'd be in favour of a campaign to get 20 somethings to turn off their cars when they are stopped | 23:36 |
| clarkb | anteaya: a lot of cars do that now automagically | 23:36 |
| fungi | okay, looks like etherpad 1.6.5 is checked out now, so i'll go ahead with the service restart | 23:36 |
| clarkb | anteaya: but not on this continent, it is more of a european thing | 23:36 |
| anteaya | I thought we had already agreed as a group to not do that but that idea seems not to have filltered down yet | 23:36 |
| anteaya | clarkb, oh I like that | 23:36 |
| clarkb | fungi: cool, I'm around to help with that if necesasry | 23:36 |
| anteaya | clarkb, it is a good european thing | 23:37 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova functional race bug 1763181 https://review.openstack.org/560640 | 23:37 |
| openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,Confirmed] https://launchpad.net/bugs/1763181 | 23:37 |
| anteaya | I don't understand why turning the ignition off is such a hard point to understand, but it seems to be | 23:37 |
| ianw | mordred: when i do "cloud.delete_image(name_or_id=sys.argv[1], wait=True)" it's not iterating through all the images first is it? | 23:38 |
| *** mriedem is now known as mriedem_afk | 23:38 | |
| fungi | #status notice The Etherpad service at https://etherpad.openstack.org/ is being restarted to pick up the latest release version; browsers should see only a brief ~1min blip before reconnecting automatically to active pads | 23:38 |
| openstackstatus | fungi: sending notice | 23:38 |
| *** HeOS has quit IRC | 23:39 | |
| ianw | mordred: cause it seems to be "GET call to image for https://iad.images.api.rackspacecloud.com/v2/images used request id req-1a8366f5-47a5-4d96-a20e-3d515bd5e447" then it keep going ... which is a bit of an issue with these hundreds of images | 23:39 |
| *** felipemonteiro_ has joined #openstack-infra | 23:39 | |
| -openstackstatus- NOTICE: The Etherpad service at https://etherpad.openstack.org/ is being restarted to pick up the latest release version; browsers should see only a brief ~1min blip before reconnecting automatically to active pads | 23:39 | |
| fungi | restart seems to have completed | 23:41 |
| anteaya | yay | 23:41 |
| openstackstatus | fungi: finished sending notice | 23:41 |
| fungi | loading pads i expect to be there still seems to work | 23:41 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova xenapi unit test bug 1763535 https://review.openstack.org/561073 | 23:42 |
| openstack | bug 1763535 in OpenStack Compute (nova) "nova.tests.unit.virt.xenapi.test_vm_utils.StreamDiskTestCase.test_non_ami intermittently fails" [High,Confirmed] https://launchpad.net/bugs/1763535 | 23:42 |
| clarkb | fungi: I've pulled up a random pad and it works | 23:42 |
| anteaya | yay working working random etherpads | 23:42 |
| *** felipemonteiro__ has quit IRC | 23:43 | |
| clarkb | https://etherpad.openstack.org/p/infra-rocky-ptg for example | 23:43 |
| anteaya | that etherpad renders for me | 23:44 |
| clarkb | https://etherpad.openstack.org/p/clarkb-test too | 23:45 |
| clarkb | I'd say it looks good | 23:45 |
| mordred | ianw: yes, it needs to do alist first. if you pass in a dict with a uuid in the id field of the dict, it will not do a list first | 23:45 |
| ianw | oh, gtk | 23:46 |
| clarkb | general heads up I'm going to be working from downtown portland tomorrow with other openstack foundation folks so may be in and out of internet conenctivity. | 23:46 |
| anteaya | clarkb, I hope you have a good day tomorrow | 23:47 |
| *** tosky has quit IRC | 23:49 | |
| ianw | mordred: like cloud.delete_image({id: i}, wait=True) ? | 23:49 |
| mordred | ianw: yup | 23:50 |
| ianw | mordred: are you sure it works :) ? | 23:51 |
| mordred | ianw: (issue being that we have no way to know if you're passing a name or an id if you just pass a string, so we do a list and check to see if the given string matches either the name or id field) | 23:51 |
| mordred | ianw: it should ... if it doesn't, it's a bug :) | 23:51 |
| mordred | ianw: oh- hrm | 23:52 |
| mordred | ianw: yup. I see a bug | 23:52 |
| mordred | sigh | 23:53 |
| *** claudiub|2 has quit IRC | 23:53 | |
| ianw | is it the hasattr line? | 23:53 |
| mordred | yup | 23:53 |
| mordred | ianw: I think it should be | 23:54 |
| mordred | if (hasattr(name_or_id, 'id') | 23:54 |
| mordred | or isinstance(name_or_id, dict) and 'id' in name_or_id): | 23:54 |
| mordred | maybe another paren in there | 23:54 |
| mordred | or (isinstance(name_or_id, dict) and 'id' in name_or_id)): | 23:54 |
| ianw | yeah, ok let me hack that in and see | 23:54 |
| mordred | sweet. sorry about that | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!