| *** dave-mccowan has joined #openstack-infra | 00:04 | |
| *** wolverineav has joined #openstack-infra | 00:04 | |
| clarkb | I'm going to copy the stash file as that suggests and see if that fixes things | 00:04 |
|---|---|---|
| *** rascasoft has joined #openstack-infra | 00:06 | |
| *** sdake_ has quit IRC | 00:08 | |
| clarkb | kadmind is now running | 00:08 |
| clarkb | I think that was it. fungi want ot try installing the package again? | 00:09 |
| *** wolverineav has quit IRC | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** wolverineav has joined #openstack-infra | 00:12 | |
| *** agopi has joined #openstack-infra | 00:15 | |
| * clarkb will try | 00:15 | |
| *** slaweq has quit IRC | 00:16 | |
| clarkb | fungi: package install still fails | 00:16 |
| *** rascasoft has quit IRC | 00:16 | |
| clarkb | fungi: any more ideas? maybe set -x again? | 00:17 |
| *** wolverineav has quit IRC | 00:17 | |
| clarkb | fwiw if we can sort this out and get puppet running sanely then I think I'd like to start work on replacing kdc01 by making kdc04 the master, booting a new server then updating dns as appropriate | 00:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 00:18 |
| clarkb | actually I think the process should be start kdc03.openstack.org and make it a slave. Then shutoff kdc01 and make kdc03 the master syncing to 04 | 00:20 |
| clarkb | but all that should happen once we have puppet happy | 00:20 |
| fungi | mmm, i can check again in a sec | 00:22 |
| clarkb | futureparser puppet run on kdc01 is happy | 00:23 |
| *** sdake has joined #openstack-infra | 00:25 | |
| corvus | YAAAAAAAAAAAAAAAAAAY!!! https://www.youtube.com/watch?v=QVPKI2j__uQ | 00:32 |
| corvus | the first speculative execution of an unpublished docker image: http://logs.openstack.org/54/637654/9/check/system-config-run-zuul-preview/0f61f28/hosts/bridge.openstack.org/ara-report/result/0ebf64bc-f6b4-4996-be94-7531dcbae9a8/ | 00:33 |
| corvus | mordred, clarkb, fungi, tobiash, dmsimard: ^ | 00:36 |
| clarkb | neat | 00:36 |
| corvus | that's running the image built in the jobs run for this change: https://review.openstack.org/637037 | 00:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use a pipeline for dib stats https://review.openstack.org/638265 | 00:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Update dib stats https://review.openstack.org/638533 | 00:37 |
| clarkb | futureparser on kdc04 doesn't seem to have chagned the behavior of puppet failing on the package install there | 00:37 |
| corvus | which is in a different repo, pulled in via depends-on | 00:37 |
| *** yamamoto has quit IRC | 00:41 | |
| *** yamamoto has joined #openstack-infra | 00:44 | |
| dmsimard | corvus: a lot of work has gone into that :) | 00:47 |
| dmsimard | ++ | 00:47 |
| *** yamamoto has quit IRC | 00:48 | |
| *** rlandy is now known as rlandy|afk | 00:49 | |
| *** wolverineav has joined #openstack-infra | 00:50 | |
| clarkb | fungi: I think I'm getting the same error you saw after setting -x | 00:52 |
| clarkb | fungi: so starting the service wasn't the only problem | 00:52 |
| *** markvoelker has joined #openstack-infra | 00:53 | |
| *** dave-mccowan has quit IRC | 00:53 | |
| *** gyee has quit IRC | 00:54 | |
| *** sthussey has quit IRC | 00:54 | |
| clarkb | I suspect it is unpacking the deb and using the postinst file there and not the one in /var/lib/dpkg/info because the set -x tracing stops once it reexecs into that | 00:55 |
| clarkb | hrm except that the command that fails is using the path that has the set -x | 00:56 |
| clarkb | this would imply the problem is in the perl script | 01:01 |
| fungi | yeah, this is why i asked about tracing perl before | 01:05 |
| clarkb | ok I set -x in the package .config file becuase that is what the perl script was trying to run before the postinstall script | 01:08 |
| clarkb | and that seems to have gotten my more interesting output | 01:08 |
| clarkb | read -r _db_internal_line | 01:08 |
| clarkb | + RET=10 krb5-admin-server/kadmind doesn't exist | 01:08 |
| fungi | oh, good call | 01:08 |
| clarkb | db_set krb5-admin-server/kadmind "$RUN_KADMIND" is what it is trying to do I think | 01:12 |
| fungi | as for the swapping kdc01 in plan, that's still running trusty, looks like, not xenial | 01:12 |
| clarkb | fungi: ya so we'll add a kdc03, make kdc03 master then delete 01 | 01:12 |
| clarkb | just have to run down a few of these other details first. Like make package installs work and also did we save the master key info anywhere? | 01:13 |
| clarkb | re debconf, it is trying to set a value why wouldn't it just create the value instead of complaining thatit doesn't exist? | 01:14 |
| clarkb | I really know little about debconf so unsure why it is unhappywith that | 01:14 |
| pabelanger | mordred: mnaser: what does it mean when openstacksdk is unable to fetch a remote profile? eg: http://paste.openstack.org/show/745671/ | 01:16 |
| clarkb | ok I think I understand. Those values have to be in the templates file? | 01:18 |
| fungi | yeah, it seems like a debconf template mismatch maybe? | 01:19 |
| clarkb | ya I think this si saying the package is broken because it is trying to set a template value that does not exit | 01:19 |
| pabelanger | mordred: mnaser: ok, it seems to be something related on vexxhost side. Downgrading to 0.21.0 for openstacksdk seems to work properly | 01:20 |
| clarkb | https://git.launchpad.net/ubuntu/+source/krb5/commit/debian/krb5-admin-server.templates?id=a1dde319af38d098771627855afc7717028b67fe is the change that broke us | 01:20 |
| clarkb | we must literally be the only people on the planet running kerberos on xenial | 01:21 |
| mnaser | pabelanger, mordred oops we broke something. | 01:21 |
| mnaser | We migrated our site today. I’ll have to get this fixed | 01:21 |
| mnaser | clarkb: we run it too! Ha | 01:21 |
| clarkb | looking at the latest version of that package I think we must be the only people running it on ubuntu? | 01:22 |
| pabelanger | mnaser: ah, it might have just happened in last 4 hours or so. this was working this afternoon :) | 01:22 |
| pabelanger | thanks for confirming | 01:22 |
| clarkb | oh or we are the only people that set /etc/default/krb5-admin-server RUN_KADMIND | 01:22 |
| clarkb | let me see if we actually need RUN_KADMIND | 01:22 |
| mnaser | yep | 01:22 |
| pabelanger | ha | 01:23 |
| pabelanger | http://grafana.openstack.org/d/nuvIH5Imk/nodepool-vexxhost?orgId=1&from=now-24h&to=now | 01:23 |
| pabelanger | I think that might explain why nodepool isn't happy | 01:23 |
| *** rascasoft has joined #openstack-infra | 01:23 | |
| pabelanger | oh, maybe not | 01:24 |
| clarkb | corvus: fungi mnaser ok I think that is it. The .config file only tries to set config if RUN_KADMIND is set to false and we set it to false on our slave node(s) | 01:24 |
| pabelanger | mnaser: clarkb: seems vexxhost-sjc1 stopped working a few days ago | 01:24 |
| clarkb | corvus: fungi mnaser is there any reason to not run kadmind on the slave node? | 01:24 |
| pabelanger | 2/19: 1800UTC it seems | 01:24 |
| clarkb | (important because it is currently running) | 01:25 |
| *** takamatsu_ has joined #openstack-infra | 01:26 | |
| *** markvoelker has quit IRC | 01:26 | |
| *** takamatsu has quit IRC | 01:26 | |
| mnaser | pabelanger: maybe same reason? | 01:26 |
| clarkb | I'm going to stop kadmind on kdc04 now | 01:28 |
| clarkb | in case that is required for safety | 01:28 |
| *** ijw has quit IRC | 01:28 | |
| fungi | yeah, your findings seem correct | 01:28 |
| fungi | and also explains why this isn't broken on 05 | 01:29 |
| *** ijw has joined #openstack-infra | 01:29 | |
| clarkb | 01 you mean? | 01:30 |
| clarkb | mnaser: you must run kadmind on your slave(s) or only have a single node master? | 01:30 |
| *** ijw has quit IRC | 01:30 | |
| clarkb | because I don't see how this packaging would haev worked for slave(s) with kadmind disabled | 01:30 |
| *** ijw has joined #openstack-infra | 01:31 | |
| fungi | i meant the master, but for some reason i thought that was 05. seems we don | 01:32 |
| fungi | 't have an 05 | 01:32 |
| fungi | i guess we never finished switching fully to xenial | 01:32 |
| clarkb | fungi: right 01 is the trusty master. 04 is the xenial slave | 01:32 |
| clarkb | also now I'm confused by systemd would even start kadmind if RUN_KADMIND is set to false | 01:33 |
| *** rascasoft has quit IRC | 01:33 | |
| clarkb | oh that only matters if restarting the service | 01:33 |
| clarkb | if you start and stop it takes you at your word | 01:33 |
| clarkb | so ya we must be the only people in the world running kerberos via debuntu packaging with more than one node | 01:36 |
| clarkb | otherwise I don't see how this could work properly | 01:36 |
| clarkb | fungi: is there a dpkg flag we can set to tell it not to configure things? | 01:36 |
| clarkb | (though I'm not sure we want that in the no previous config case, but maybe puppet already handles that for us?) | 01:37 |
| fungi | mmm, we can disable the initscript with update-rc2.d i think? | 01:37 |
| fungi | er, update-rc.d | 01:37 |
| clarkb | fungi: you tried noninteractive right? | 01:38 |
| clarkb | and that didn't change it? | 01:38 |
| fungi | yeah, tried noninteractive | 01:38 |
| clarkb | I think because it will still try to set that value automaticallly right than query for it | 01:38 |
| clarkb | i'm going to remove my set -x's now | 01:38 |
| fungi | that just prevents debconf from prompting you with anything | 01:38 |
| fungi | not sure if update-rc.d is relevant for systemd either | 01:39 |
| *** ijw has quit IRC | 01:39 | |
| clarkb | no I think we'd need to do the equivalent systemd thing | 01:39 |
| clarkb | systemctl disable krb5-admin-server or somesuch | 01:39 |
| fungi | but maybe disabling the krb5-admin-server is preferred over setting RUN_KADMIND=false | 01:39 |
| clarkb | ya that is possible | 01:40 |
| clarkb | also maybe we can get away with running kadmind on all the nodes? the docs don't seem to say tht anywhere though | 01:40 |
| *** yamamoto has joined #openstack-infra | 01:41 | |
| *** ijw has joined #openstack-infra | 01:41 | |
| *** sdake has quit IRC | 01:44 | |
| mnaser | clarkb: freeipa makes my life easy | 01:45 |
| clarkb | wow even better is there is a systemd unit file for this service | 01:46 |
| clarkb | which seems to ignore the defaults file | 01:46 |
| *** sdake has joined #openstack-infra | 01:46 | |
| clarkb | so maybe that is how you are expected to do things now. Ignore the defaults file and the sysv init script and enable/disable via systemctl | 01:47 |
| fungi | seems likely | 01:49 |
| fungi | enable flags in defaults files sourced by initscripts were always seen as a somewhat hacky solution | 01:49 |
| clarkb | I'm working on a patch to puppet-kerberos | 01:51 |
| clarkb | then dinner | 01:51 |
| fungi | i have to get back to working on the walls but will take a look at whatever you push up for this | 01:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-kerberos master: Workaround broken ubuntu packaging https://review.openstack.org/638570 | 01:57 |
| clarkb | I think ^ will end up working around this for us | 01:57 |
| clarkb | fungi: also this seems like the sort of thing that package linters should catch? | 02:00 |
| *** wolverineav has quit IRC | 02:01 | |
| fungi | maybe, but honestly i don't know to what extent ubuntu enforces that and whether they may have ganked some of this from earlier debian packages | 02:02 |
| clarkb | fungi: it appears it is ported from debian packaging but it also seems like they just copy that over from debian? | 02:03 |
| clarkb | I wouldn't be surprised if debian has the same bug | 02:03 |
| clarkb | yes debian has the same bug in it | 02:04 |
| clarkb | in sid | 02:04 |
| fungi | fun | 02:05 |
| fungi | worth reporting a bug i suppose | 02:05 |
| clarkb | ya I'll file one in the morning if I can figure out debians bug system | 02:05 |
| clarkb | then I can link to that from a launchpad bug | 02:05 |
| fungi | debian's bug system can be interfaced entirely over smtp | 02:06 |
| fungi | though the reportbug utility is usually preferred for putting together the report itself | 02:06 |
| clarkb | ya I always get confused by it | 02:06 |
| clarkb | also how crnaky will people be if I say "didn't actually install on debian but this happens on ubuntu and your package files aren't any different" | 02:07 |
| *** bhavikdbavishi has joined #openstack-infra | 02:08 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** hamzy has joined #openstack-infra | 02:15 | |
| *** slaweq has quit IRC | 02:15 | |
| *** bgmccollum has quit IRC | 02:15 | |
| clarkb | also note that the workaround change will need manual intervention on the server to get unstuck | 02:17 |
| clarkb | we requier the package be installed before updating the defaults file but the package isn't installing :) | 02:18 |
| clarkb | thats fine I can remove the RUN_KADMIND entry from the defaults file once merged | 02:18 |
| *** bgmccollum has joined #openstack-infra | 02:18 | |
| clarkb | and now I am off for the evening. I'll pick this up tomorrow. ianw maybe you would like to review the puppet-kerberos change above? I'm happy to +A if you just want to review it and not babysit | 02:19 |
| ianw | heh yes just did, was following along :) | 02:19 |
| clarkb | thank you! | 02:20 |
| *** markvoelker has joined #openstack-infra | 02:23 | |
| *** wolverineav has joined #openstack-infra | 02:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Update dib stats https://review.openstack.org/638533 | 02:26 |
| fungi | we could just manually purge the package and let puppet set it back up? | 02:28 |
| *** wolverineav has quit IRC | 02:29 | |
| *** rfolco|rover has quit IRC | 02:30 | |
| *** wolverineav has joined #openstack-infra | 02:31 | |
| *** sdake has quit IRC | 02:32 | |
| *** wolverineav has quit IRC | 02:33 | |
| *** ijw has quit IRC | 02:35 | |
| *** sdake has joined #openstack-infra | 02:35 | |
| *** sdake_ has joined #openstack-infra | 02:38 | |
| *** sdake has quit IRC | 02:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add nodepool-dib dashboard https://review.openstack.org/638325 | 02:40 |
| *** ccamacho has quit IRC | 02:48 | |
| *** rascasoft has joined #openstack-infra | 02:56 | |
| *** markvoelker has quit IRC | 02:57 | |
| *** yamamoto has quit IRC | 02:57 | |
| *** whoami-rajat has joined #openstack-infra | 02:57 | |
| *** yamamoto has joined #openstack-infra | 03:02 | |
| *** sdake_ has quit IRC | 03:03 | |
| *** hwoarang has quit IRC | 03:03 | |
| *** psachin has joined #openstack-infra | 03:04 | |
| *** rascasoft has quit IRC | 03:05 | |
| *** hwoarang has joined #openstack-infra | 03:07 | |
| *** sdake has joined #openstack-infra | 03:10 | |
| *** apetrich has quit IRC | 03:16 | |
| *** janki has joined #openstack-infra | 03:20 | |
| *** emccormick has quit IRC | 03:23 | |
| *** bhavikdbavishi has quit IRC | 03:30 | |
| *** sdake has quit IRC | 03:33 | |
| *** emccormick has joined #openstack-infra | 03:42 | |
| *** udesale has joined #openstack-infra | 03:53 | |
| *** markvoelker has joined #openstack-infra | 03:54 | |
| *** rascasoft has joined #openstack-infra | 03:55 | |
| *** yamamoto has quit IRC | 03:56 | |
| *** yamamoto has joined #openstack-infra | 03:59 | |
| *** rascasoft has quit IRC | 04:00 | |
| *** yamamoto has quit IRC | 04:05 | |
| *** diablo_rojo has quit IRC | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| mordred | pabelanger, mnaser: I +A the sdk change for the error message fix (in gate now) - but yeah - if that file goes away now, sadness will ensue | 04:13 |
| *** anteaya has quit IRC | 04:14 | |
| *** hwoarang has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:16 | |
| *** hwoarang has joined #openstack-infra | 04:17 | |
| *** wolverineav has joined #openstack-infra | 04:20 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:20 | |
| *** rlandy|afk is now known as rlandy | 04:21 | |
| *** wolverineav has quit IRC | 04:24 | |
| *** markvoelker has quit IRC | 04:27 | |
| *** ramishra has joined #openstack-infra | 04:48 | |
| *** lpetrut has joined #openstack-infra | 04:49 | |
| *** rascasoft has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** yamamoto has joined #openstack-infra | 05:12 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Update nodepool dib stats https://review.openstack.org/638583 | 05:14 |
| *** slaweq has quit IRC | 05:15 | |
| *** yamamoto has quit IRC | 05:16 | |
| *** rascasoft has quit IRC | 05:18 | |
| *** markvoelker has joined #openstack-infra | 05:24 | |
| *** lpetrut has quit IRC | 05:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add nodepool-dib dashboard https://review.openstack.org/638325 | 05:26 |
| *** gmann has quit IRC | 05:28 | |
| *** kjackal has joined #openstack-infra | 05:32 | |
| *** ociuhandu has joined #openstack-infra | 05:36 | |
| *** yamamoto has joined #openstack-infra | 05:37 | |
| *** ociuhandu has quit IRC | 05:40 | |
| *** hwoarang has quit IRC | 05:46 | |
| *** hwoarang has joined #openstack-infra | 05:48 | |
| *** kjackal has quit IRC | 05:51 | |
| *** kjackal has joined #openstack-infra | 05:52 | |
| *** lpetrut has joined #openstack-infra | 05:54 | |
| *** kjackal has quit IRC | 05:56 | |
| *** markvoelker has quit IRC | 05:58 | |
| *** hwoarang has quit IRC | 06:27 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add Redirect options to static https vhosts https://review.openstack.org/638527 | 06:27 |
| *** calebb has quit IRC | 06:27 | |
| *** hwoarang has joined #openstack-infra | 06:28 | |
| *** ykarel|pto has joined #openstack-infra | 06:38 | |
| *** ykarel|pto is now known as ykarel | 06:40 | |
| *** yamamoto has quit IRC | 06:50 | |
| *** yamamoto has joined #openstack-infra | 06:50 | |
| *** kjackal has joined #openstack-infra | 06:53 | |
| *** hwoarang has quit IRC | 06:54 | |
| *** markvoelker has joined #openstack-infra | 06:55 | |
| *** sdake has joined #openstack-infra | 06:59 | |
| *** quiquell|off is now known as quiquell | 06:59 | |
| *** bhavikdbavishi has quit IRC | 07:01 | |
| *** hwoarang has joined #openstack-infra | 07:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix opensuse pip-and-virtualenv https://review.openstack.org/638594 | 07:05 |
| *** kjackal has quit IRC | 07:08 | |
| *** kjackal has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Optionally disable disk_limit_per_job https://review.openstack.org/638596 | 07:13 |
| *** slaweq has quit IRC | 07:15 | |
| *** stakeda has joined #openstack-infra | 07:20 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Remove periodic-package-stackviz-element job https://review.openstack.org/638598 | 07:23 |
| *** rascasoft has joined #openstack-infra | 07:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tox-py35-on-zuul from zuul-jobs https://review.openstack.org/638525 | 07:26 |
| *** jtomasek has joined #openstack-infra | 07:27 | |
| *** dpawlik has quit IRC | 07:28 | |
| *** markvoelker has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-infra | 07:29 | |
| chandankumar | AJaeger: Hello | 07:31 |
| chandankumar | AJaeger: I need to remove this legacy job https://github.com/openstack/stackviz/blob/master/.zuul.yaml#L10 and move it to proper publish to pypi job | 07:31 |
| *** sdake has quit IRC | 07:33 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Remove periodic-package-stackviz-element job https://review.openstack.org/638598 | 07:34 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add publish to pypi job for stackviz https://review.openstack.org/638602 | 07:38 |
| chandankumar | AJaeger: ^^ I have proposed the above patches, feel free to take a look. thanks :-) | 07:39 |
| *** dpawlik has joined #openstack-infra | 07:39 | |
| *** e0ne has quit IRC | 07:43 | |
| *** e0ne has joined #openstack-infra | 07:44 | |
| AJaeger | chandankumar: I'm confused | 07:44 |
| *** aojea has joined #openstack-infra | 07:44 | |
| *** e0ne has quit IRC | 07:44 | |
| AJaeger | the legacy job is an npm job that is run daily - you replace it iwth a python publish job publishing to pypi. | 07:45 |
| AJaeger | chandankumar: I commented - if this is correct, the project-config change needs far more explanation... | 07:46 |
| *** ykarel is now known as ykarel|lunch | 07:46 | |
| *** emccormick has quit IRC | 07:49 | |
| *** e0ne has joined #openstack-infra | 07:50 | |
| chandankumar | AJaeger: Currently there are two tarballs getting published related to stackviz. | 07:51 |
| chandankumar | http://tarballs.openstack.org/package-stackviz-element/ | 07:51 |
| chandankumar | and http://tarballs.openstack.org/stackviz/ | 07:51 |
| chandankumar | First one is the correct tarball it contains stackviz html file and python binary but | 07:51 |
| chandankumar | second one contains only pypi which is not usable. | 07:51 |
| chandankumar | In RDO side, while package, I have used second one. | 07:51 |
| chandankumar | AJaeger: First one getting generated from periodic-package-stackviz-element job but in the end | 07:51 |
| chandankumar | it is pubished as a python package. it is causing confusion due to two tarballs, so proposed to | 07:51 |
| chandankumar | remove it | 07:51 |
| chandankumar | AJaeger: or may be I am doing something wrong | 07:52 |
| *** ginopc has joined #openstack-infra | 07:53 | |
| *** yamamoto has quit IRC | 07:56 | |
| *** yamamoto has joined #openstack-infra | 07:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:59 | |
| *** kopecmartin|off is now known as kopecmartin | 08:00 | |
| *** yamamoto has quit IRC | 08:02 | |
| *** slaweq has joined #openstack-infra | 08:08 | |
| *** yamamoto has joined #openstack-infra | 08:09 | |
| *** ykarel|lunch is now known as ykarel | 08:10 | |
| *** e0ne has quit IRC | 08:10 | |
| *** tkajinam has quit IRC | 08:13 | |
| *** yamamoto has quit IRC | 08:13 | |
| AJaeger | chandankumar: look at the timestamps, the one folder is from 2017! | 08:15 |
| AJaeger | chandankumar: I have no clue about stackviz, I just see that you change publishing and the commit message does not explain to me why the change you do is the right one ;( | 08:16 |
| *** sdake has joined #openstack-infra | 08:18 | |
| chandankumar | AJaeger: thanks, I think I got the solution in rdo packaging side to use source from git and rebuild npm and reuse it in rdo packaging | 08:18 |
| chandankumar | i will abandon the reviews | 08:18 |
| *** yamamoto has joined #openstack-infra | 08:22 | |
| *** markvoelker has joined #openstack-infra | 08:25 | |
| *** apetrich has joined #openstack-infra | 08:26 | |
| *** jpich has joined #openstack-infra | 08:51 | |
| *** jpich has quit IRC | 08:52 | |
| *** jpich has joined #openstack-infra | 08:52 | |
| *** sshnaidm is now known as sshnaidm|off | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 08:58 | |
| *** markvoelker has quit IRC | 08:59 | |
| *** ykarel is now known as ykarel|lunch | 08:59 | |
| *** jpena|off is now known as jpena | 09:02 | |
| AJaeger | chandankumar: if the periodic job is broken, we can remove it as well - we don't do this for any other repo... | 09:09 |
| *** ociuhandu has joined #openstack-infra | 09:10 | |
| chandankumar | AJaeger: is there a way to just run npm build and then run python setup.py sdist together | 09:10 |
| chandankumar | AJaeger: currently I am debugging it | 09:10 |
| AJaeger | chandankumar: I'm not aware of anything - but you can search using codesearch.openstack.org | 09:11 |
| *** kjackal has quit IRC | 09:12 | |
| *** kjackal has joined #openstack-infra | 09:17 | |
| *** panda|ruck|off is now known as panda|ruck | 09:24 | |
| *** sdake has quit IRC | 09:26 | |
| *** yamamoto has quit IRC | 09:28 | |
| *** jaosorior has quit IRC | 09:33 | |
| *** jaosorior has joined #openstack-infra | 09:35 | |
| *** electrofelix has joined #openstack-infra | 09:36 | |
| openstackgerrit | Jakub Bielecki proposed openstack-infra/nodepool master: doc bugfix for static provider https://review.openstack.org/637518 | 09:40 |
| *** ykarel|lunch is now known as ykarel | 09:41 | |
| *** stakeda has quit IRC | 09:47 | |
| *** stephenfin is now known as finucannot | 09:48 | |
| *** markvoelker has joined #openstack-infra | 09:56 | |
| *** dtantsur is now known as dtantsur|brb | 10:01 | |
| *** gfidente has joined #openstack-infra | 10:03 | |
| *** takamatsu_ has quit IRC | 10:04 | |
| *** takamatsu has joined #openstack-infra | 10:05 | |
| *** yamamoto has joined #openstack-infra | 10:08 | |
| *** yamamoto has quit IRC | 10:13 | |
| *** e0ne has joined #openstack-infra | 10:19 | |
| *** gfidente has quit IRC | 10:22 | |
| *** takamatsu_ has joined #openstack-infra | 10:23 | |
| *** helenaAM has joined #openstack-infra | 10:23 | |
| *** takamatsu has quit IRC | 10:24 | |
| *** gfidente has joined #openstack-infra | 10:26 | |
| *** markvoelker has quit IRC | 10:28 | |
| *** luizbag has joined #openstack-infra | 10:29 | |
| *** rcernin has quit IRC | 10:31 | |
| *** gfidente has quit IRC | 10:37 | |
| *** takamatsu_ has quit IRC | 10:48 | |
| *** takamatsu has joined #openstack-infra | 10:49 | |
| *** ccamacho has joined #openstack-infra | 10:54 | |
| *** ccamacho has quit IRC | 10:54 | |
| *** takamatsu has quit IRC | 10:54 | |
| *** shardy has joined #openstack-infra | 10:54 | |
| *** takamatsu has joined #openstack-infra | 10:58 | |
| *** udesale has quit IRC | 10:59 | |
| *** shardy has quit IRC | 11:06 | |
| *** sdake has joined #openstack-infra | 11:11 | |
| *** tosky has joined #openstack-infra | 11:12 | |
| aspiers | is the 'recheck' zuul directive documented anywhere? I can't find it if so | 11:18 |
| aspiers | and is it possible to just recheck an individual job, not the whole lot? | 11:18 |
| *** jlibosva has joined #openstack-infra | 11:23 | |
| *** markvoelker has joined #openstack-infra | 11:25 | |
| frickler | aspiers: the answer to the second question is "intentionally no". I'll check for docs in a minut | 11:26 |
| tosky | aspiers: documented at least here: https://docs.openstack.org/doc-contrib-guide/quickstart/first-timers.html | 11:26 |
| tosky | well, at least mentioned | 11:26 |
| aspiers | tosky: thanks! | 11:26 |
| aspiers | frickler: interesting, what's the intention with that? | 11:26 |
| frickler | aspiers: avoiding getting patches to pass unstable jobs more easily | 11:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 11:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade some servers to puppet 4 https://review.openstack.org/634726 | 11:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade git01.openstack.org to puppet 4 https://review.openstack.org/634727 | 11:27 |
| aspiers | frickler: you mean rather than fixing the instability? | 11:28 |
| frickler | tosky: oh, that one still mentions console.html, we should update it | 11:28 |
| tosky | aspiers: aka: recheck every job independently until you have everything passing | 11:28 |
| frickler | aspiers: tosky: exactly | 11:28 |
| aspiers | Hrm. It's a good intention but I'm not sure how effective it is | 11:29 |
| aspiers | Maybe it works with some people | 11:29 |
| frickler | aspiers: if you intentionally want to run only a single job against your patch multiple times, you can modify zuul.yaml to only run that job | 11:29 |
| aspiers | frickler: No I don't | 11:29 |
| *** bhavikdbavishi has quit IRC | 11:30 | |
| tosky | I think I created a story to request the possibility to recheck a single experimental job but without having it voting | 11:30 |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 11:30 |
| aspiers | https://review.openstack.org/#/c/633855/ got V-1 due to some random instability in nova-next which I don't have the first clue how to fix. So my only choice currently is to waste a huge bunch of CI resources by rechecking 25 jobs just because a single voting one failed. This does not make sense to me. | 11:31 |
| aspiers | Especially considering on the next recheck, any number of other partially stable jobs in this list could also fail for reasons unrelated to my change | 11:32 |
| frickler | that's why folk like mriedem give so high priority to fixing unstable jobs | 11:33 |
| aspiers | If you have 25 jobs of which 5 randomly fail 10% of the time, the probability of getting V+1 is less than 60% | 11:34 |
| *** yamamoto has quit IRC | 11:34 | |
| aspiers | frickler: And of course we're all grateful for those heroes. But many contributors don't have enough knowledge to help with the instabilities, so I'm wondering if burning CI resources through unnecessary rechecks is the right approach. | 11:35 |
| *** auristor has quit IRC | 11:35 | |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 11:36 |
| aspiers | Anyway, just fancied providing some food for thought ;-) No need for action right now. | 11:36 |
| *** kjackal has quit IRC | 11:40 | |
| jlibosva | hello, I have some issue I can't tackle with zuul. I think somebody more experienced with zuul than I am can spot the problem quickly. I backported a job from master to rocky but the job doesn't get triggered in rocky, it's just not started by zuul. I tried to define branches and some other stuff but can't make it running. Can anybody help, please? | 11:42 |
| frickler | jlibosva: do you have a pointer to a review? | 11:43 |
| jlibosva | frickler: sure: https://review.openstack.org/#/c/638427 | 11:43 |
| jlibosva | frickler: maybe best would be to look at PS1 - the rest of patchsets are rather experiments | 11:44 |
| jlibosva | the same job is defined in master branch so I'm not sure how to make it running on rocky, if I need to define it in master with branches attribute or I need a backport ... | 11:45 |
| *** auristor has joined #openstack-infra | 11:45 | |
| *** priteau has joined #openstack-infra | 11:46 | |
| *** auristor has quit IRC | 11:49 | |
| *** gmann has joined #openstack-infra | 11:49 | |
| frickler | jlibosva: hmm, I don't see anything obvious, seems to me like the backport to rocky in PS1 should just work. I'm trying to check zuul logs now | 11:50 |
| jlibosva | frickler: thanks for looking into it. so is it fine the job names and job templates have the same name as master? i know zuul configuratoin is global | 11:51 |
| *** auristor has joined #openstack-infra | 11:51 | |
| AJaeger | jlibosva: you can enable debugging, see https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug | 11:54 |
| frickler | jlibosva: should be the same thing that happens when master is branched e.g. to stable/stein, that also supposed to just work without any additional action | 11:54 |
| AJaeger | Add that to check pipeline and then once all jobs passed, it will give you debug output - that might help pinpoint the problem | 11:54 |
| jlibosva | AJaeger: thanks, I will try it out | 11:54 |
| AJaeger | toogle CI output to see it in web ui once check was run | 11:55 |
| frickler | jlibosva: 2019-02-22 10:28:31,682 DEBUG zuul.layout: No matching parents for job tempest-multinode-full and change <Change 0x7fb78e31e860 openstack/networking-ansible 638427,2> | 11:57 |
| frickler | jlibosva: so devstack defines tempest-multinode-full only in master it seems | 11:57 |
| jlibosva | frickler: I thought that comes from tempest and that is branchless | 11:57 |
| *** markvoelker has quit IRC | 11:59 | |
| frickler | jlibosva: ah, yes. but it has a branch restriction: http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n201 | 11:59 |
| *** janki has quit IRC | 12:01 | |
| *** kjackal has joined #openstack-infra | 12:02 | |
| *** sdake has quit IRC | 12:03 | |
| *** AJaeger has quit IRC | 12:04 | |
| *** sdake_ has joined #openstack-infra | 12:04 | |
| jlibosva | frickler: that was the first thing I tried :) I tried to override it in PS2. Also if it doesn't find the parent, then it won't even get to that restriction, will it? | 12:05 |
| *** rfolco|rover has joined #openstack-infra | 12:06 | |
| frickler | jlibosva: I don't think that you can override that restriction in your job because it is tagged to the parent | 12:07 |
| frickler | jlibosva: so either ask in #-qa whether that job could be enabled for stable/rocky now, or copy the complete job description from tempest as a workaround. or maybe some zuul specialist like corvus comes up with a more clever solution ;) | 12:08 |
| jlibosva | frickler: will do, thanks for your help, I've been staring at it way too long :) | 12:09 |
| *** dpawlik has quit IRC | 12:16 | |
| *** udesale has joined #openstack-infra | 12:16 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** AJaeger has joined #openstack-infra | 12:17 | |
| *** Tengu has quit IRC | 12:17 | |
| *** Tengu has joined #openstack-infra | 12:17 | |
| *** gfidente has joined #openstack-infra | 12:20 | |
| *** dpawlik has joined #openstack-infra | 12:23 | |
| *** EmilienM is now known as EvilienM | 12:27 | |
| *** priteau has quit IRC | 12:31 | |
| *** dtantsur|brb is now known as dtantsur | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:35 | |
| *** gfidente has quit IRC | 12:38 | |
| *** priteau has joined #openstack-infra | 12:39 | |
| *** ccamacho has joined #openstack-infra | 12:39 | |
| *** gfidente has joined #openstack-infra | 12:40 | |
| *** roman_g has joined #openstack-infra | 12:41 | |
| *** roman_g has quit IRC | 12:46 | |
| *** markvoelker has joined #openstack-infra | 12:56 | |
| *** ociuhandu has quit IRC | 13:01 | |
| *** jcoufal has joined #openstack-infra | 13:01 | |
| *** trown|outtypewww is now known as trown | 13:03 | |
| *** kgiusti has joined #openstack-infra | 13:08 | |
| gmann | frickler: jlibosva i am trying it to make it for stable branches but facing few issue - https://review.openstack.org/#/c/620582/ | 13:08 |
| *** auristor has quit IRC | 13:09 | |
| *** mriedem has joined #openstack-infra | 13:10 | |
| *** panda|ruck is now known as panda|lunch | 13:10 | |
| *** auristor has joined #openstack-infra | 13:11 | |
| *** dave-mccowan has joined #openstack-infra | 13:18 | |
| *** florianf has joined #openstack-infra | 13:19 | |
| *** quiquell is now known as quiquell|off | 13:22 | |
| *** udesale has quit IRC | 13:25 | |
| *** markvoelker has quit IRC | 13:28 | |
| *** jlibosva has quit IRC | 13:30 | |
| *** rlandy has joined #openstack-infra | 13:34 | |
| *** psachin has quit IRC | 13:35 | |
| *** hamzy has quit IRC | 13:35 | |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 13:38 |
| *** hamzy has joined #openstack-infra | 13:40 | |
| *** udesale has joined #openstack-infra | 13:41 | |
| *** dave-mccowan has quit IRC | 13:43 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-infra | 13:43 | |
| *** dave-mccowan has joined #openstack-infra | 13:43 | |
| *** agopi has quit IRC | 13:44 | |
| *** ccamacho has quit IRC | 13:50 | |
| *** ccamacho has joined #openstack-infra | 13:51 | |
| *** sdake_ has quit IRC | 13:51 | |
| *** agopi has joined #openstack-infra | 13:54 | |
| *** florianf has quit IRC | 13:57 | |
| *** agopi_ has joined #openstack-infra | 13:58 | |
| *** agopi has quit IRC | 14:00 | |
| *** sthussey has joined #openstack-infra | 14:03 | |
| *** dave-mccowan has quit IRC | 14:05 | |
| *** ekultails has joined #openstack-infra | 14:06 | |
| mnaser | would anyone happen to know why mirror01.ca-ymq-1.vexxhost.openstack.org still exists? | 14:06 |
| mnaser | it looks like mirror02 exists and it's what we're pointing to | 14:06 |
| mnaser | i know we brought up mirror02 to use a new flavor and we switched to it | 14:07 |
| mnaser | mirror.ca-ymq-1.vexxhost.openstack.org is a CNAME to mirror02.ca-ymq-1.vexxhost.openstack.org | 14:07 |
| *** jpena|lunch is now known as jpena | 14:13 | |
| *** panda|lunch is now known as panda | 14:13 | |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| *** sdake has joined #openstack-infra | 14:18 | |
| *** panda is now known as panda|rcuk | 14:21 | |
| *** panda|rcuk is now known as panda|ruck | 14:21 | |
| *** dpawlik has quit IRC | 14:23 | |
| *** priteau has quit IRC | 14:25 | |
| *** markvoelker has joined #openstack-infra | 14:25 | |
| *** eharney has joined #openstack-infra | 14:25 | |
| *** jamesmcarthur has quit IRC | 14:26 | |
| fungi | likely someone missed cleaning it up. happy to delete it now | 14:27 |
| *** cmurphy is now known as cmorpheus | 14:27 | |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 14:28 |
| fungi | i have vague memories there was something weird we were troubleshooting with it, so maybe whatever that was never got brought to a conclusion when we got sidetracked by some other fire | 14:28 |
| fungi | #status log deleted old mirror01.ca-ymq-1.vexxhost.openstack.org server, long since replaced by mirror02 | 14:30 |
| openstackstatus | fungi: finished logging | 14:30 |
| fungi | i also see an available cinder volume there named nb03.openstack.org/main01 | 14:31 |
| fungi | but nb03 doesn't exist there | 14:31 |
| fungi | cleaning up that unused volume while i'm at it | 14:31 |
| fungi | #status log deleted unused nb03.openstack.org/main01 cinder volume from vexxhost ca-ymq-1 | 14:32 |
| openstackstatus | fungi: finished logging | 14:32 |
| fungi | mnaser: thanks for pointing that out! | 14:33 |
| mnaser | fungi: thank you for the cleanup | 14:33 |
| fungi | it's the least i can do | 14:34 |
| *** dave-mccowan has joined #openstack-infra | 14:38 | |
| *** bnemec is now known as beekneemech | 14:39 | |
| *** priteau has joined #openstack-infra | 14:39 | |
| *** jamesmcarthur has joined #openstack-infra | 14:40 | |
| *** sdake has quit IRC | 14:43 | |
| *** sdake has joined #openstack-infra | 14:46 | |
| *** dklyle has quit IRC | 14:46 | |
| *** david-lyle has joined #openstack-infra | 14:46 | |
| mnaser | mordred, pabelanger: https://vexxhost.com/.well-known/openstack/api is back | 14:48 |
| mnaser | sorry about that, we moved where our site is hosted and i guess that was missed (and no one really monitored that) | 14:48 |
| *** luizbag has quit IRC | 14:48 | |
| mordred | mnaser: yay for new features breaking things! | 14:51 |
| mordred | mnaser: we should really communicate the support for that out more wider - and I should probably make a gophercloud patch | 14:51 |
| mnaser | mordred: yeah, it's super neat. we should maybe work with something like keystone to be able to add it there? that way deployment tools can do this natively | 14:52 |
| mnaser | so openstack:5000/.well-known/openstack/api | 14:52 |
| mnaser | or maybe each service should expose their own stuff.. i dunno | 14:53 |
| mordred | mnaser: yeah ... although openstack:5000 would make the well-known part be weird | 14:53 |
| mnaser | yeah that's a bit redundant | 14:53 |
| mordred | oh -no, each service should definitely not :) | 14:53 |
| mordred | mnaser: BUT - I agree, it would be nice for keystone to support it or something | 14:53 |
| mordred | just not sure what the or something is :) | 14:53 |
| mnaser | solving problemsss | 14:54 |
| mordred | so much problemssss | 14:54 |
| mordred | mnaser: maybe step one is figuring out how to add support to devstack | 14:55 |
| *** bhavikdbavishi has joined #openstack-infra | 14:55 | |
| mordred | mnaser: one of the tricky bits is that like, with your deploy, it's not actually in a location relative to the openstack install - it's in a location relative to your marketing website | 14:55 |
| mnaser | mordred: yeah that's what i was thinking, putting it in keystone is a bit redundant | 14:56 |
| *** efried is now known as fried_rice | 14:56 | |
| mnaser | thats why i thought if each service provided it's own thing like "hi im glance and i use raw only" "hi im nova and i support bfv only" | 14:56 |
| mordred | mnaser: so for smaller deploys, or intranets, or whatnot, serving it from keystone or something wouldn't be a bad idea - but for things with sane dns, it wants to be deployed to some other location | 14:56 |
| mordred | mnaser: oh - yeah - but then we get in to the "support service feature discovery in each service" | 14:57 |
| mordred | mnaser: also - that would require deployers to upgrade - whereas even rackspace could deploy a .well-known/openstack/api json file manually to https://rackspace.com if they chose to | 14:57 |
| mnaser | true | 14:58 |
| mordred | it's a tricky question isn't it? | 14:58 |
| mnaser | but maybe that way we can get people to upgrade?!?! | 14:58 |
| mnaser | :P | 14:58 |
| *** markvoelker has quit IRC | 14:59 | |
| * mordred hands mnaser more coffee | 14:59 | |
| mnaser | :) | 15:00 |
| mnaser | it's very obviously friday | 15:00 |
| *** ccamacho has quit IRC | 15:01 | |
| * mordred hands mnaser more friday | 15:03 | |
| *** ykarel is now known as ykarel|away | 15:10 | |
| *** bhavikdbavishi has quit IRC | 15:14 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:15 | |
| *** rh-jelabarre has joined #openstack-infra | 15:17 | |
| *** udesale has quit IRC | 15:18 | |
| *** sdake has quit IRC | 15:21 | |
| JpMaxMan | corvus: just checking to see if you have any updates on the zuul preview for the netlify gerrit integration? | 15:23 |
| *** sdake has joined #openstack-infra | 15:23 | |
| corvus | JpMaxMan: we have all the code written and the deployment tooling is ready to go. i just got sucked into something that ended up taking longer than i expected, so i haven't actually launched the service yet | 15:24 |
| corvus | JpMaxMan: i'm hoping to do that today | 15:24 |
| JpMaxMan | corvus: ok great - thanks for the update... was just on a call and was being asked :) | 15:25 |
| fungi | i'm about to disappear so i can go exchange pleasantries with a tax accountant for the rest of my morning, but can hopefully help out after lunch | 15:26 |
| *** priteau has quit IRC | 15:27 | |
| mordred | corvus: that turned in to a deeper rabbit hole than we thought didn't it? | 15:32 |
| corvus | we learned so many learnings. | 15:32 |
| fungi | it was a veritable spelunking expedition | 15:33 |
| *** david-lyle is now known as dklyle | 15:33 | |
| *** Vadmacs has joined #openstack-infra | 15:34 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard-webclient master: Most recently updated projects should be at top of stories list https://review.openstack.org/638690 | 15:36 |
| *** dklyle has quit IRC | 15:43 | |
| *** david-lyle has joined #openstack-infra | 15:43 | |
| fungi | okay, heading out now but should return in a few hours | 15:46 |
| *** yamamoto has quit IRC | 15:51 | |
| *** ykarel|away has quit IRC | 15:55 | |
| *** markvoelker has joined #openstack-infra | 15:56 | |
| *** kashyap has joined #openstack-infra | 15:58 | |
| kashyap | Hey folks, is AppArmour used by default on all the CI guests? | 15:58 |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 16:02 |
| *** kopecmartin is now known as kopecmartin|off | 16:06 | |
| *** raissa has joined #openstack-infra | 16:10 | |
| *** e0ne has quit IRC | 16:11 | |
| *** ykarel|away has joined #openstack-infra | 16:14 | |
| kashyap | clarkb: ^ When you get a moment, do you know the answer to AppArmour question above? | 16:18 |
| mordred | kashyap: whatever is used in the base os of the guests is used - so I'd expect apparmour on ubuntu and selinux on rh. I do not believe we do anything specific in the nodes to alter stuff like that | 16:18 |
| *** aojea has quit IRC | 16:18 | |
| *** gfidente has quit IRC | 16:19 | |
| kashyap | mordred: Yeah, noted. I'd just like to verify | 16:19 |
| *** e0ne has joined #openstack-infra | 16:20 | |
| kashyap | Trying to debug a somewhat silent failure (https://bugs.launchpad.net/nova/+bug/1817324) | 16:20 |
| openstack | Launchpad bug 1817324 in OpenStack Compute (nova) "Intermittent "Failed to start libvirt guest: libvirt.libvirtError: monitor socket did not show up: No such file or directory" failures in the gate" [Undecided,Confirmed] | 16:20 |
| kashyap | And the potential cause is due to AppArmour denials, and that is difficult to debug that in OpenStack envs... | 16:21 |
| pabelanger | mnaser: yay, thanks | 16:25 |
| *** yamamoto has joined #openstack-infra | 16:28 | |
| *** markvoelker has quit IRC | 16:28 | |
| *** agopi_ is now known as agopi | 16:29 | |
| *** roman_g has joined #openstack-infra | 16:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove buildset registry job https://review.openstack.org/638706 | 16:30 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add docker image jobs https://review.openstack.org/638707 | 16:30 |
| clarkb | kashyap: you could grab the audit log | 16:31 |
| *** e0ne has quit IRC | 16:31 | |
| clarkb | or disable apparmor and check if behavior changes | 16:32 |
| *** gfidente has joined #openstack-infra | 16:32 | |
| *** rossella_s has quit IRC | 16:32 | |
| *** yamamoto has quit IRC | 16:32 | |
| kashyap | clarkb: Where is the audit.log here? -- http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/ | 16:33 |
| clarkb | kashyap: it may not be logged yet. But you could modify the job to grab it. Give me a few to dig up where to do that | 16:34 |
| kashyap | That'd be great | 16:35 |
| *** ricolin has joined #openstack-infra | 16:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Build docker image https://review.openstack.org/637037 | 16:37 |
| *** ricolin has quit IRC | 16:37 | |
| clarkb | kashyap: https://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/capture-system-logs/tasks/main.yaml | 16:37 |
| clarkb | I've approved the puppet-kerberos package brokeness workaround and will keep an eye on the two kdcs. Then I think I need to make sure the krb5kdc process is running on kdc04 after puppet runs and if so I think we have a happy xenial kdc. Next step after that is booting a xenial kdc03 as a slave. Then we'll sort out a switch of the master to kdc03 probably sometime early next week | 16:39 |
| mordred | clarkb: ++ | 16:39 |
| *** gyee has joined #openstack-infra | 16:40 | |
| clarkb | also I probably won't boot that as kdc03.opendev.org since the kerberos realm is openstack.org? | 16:40 |
| clarkb | switching the realm over to opendev seems like a future exercise | 16:41 |
| kashyap | clarkb: /me clicks | 16:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: run-base: configure docker mirrors on all hosts in CI https://review.openstack.org/638200 | 16:42 |
| clarkb | kashyap: you should be able to add a file copy of /var/log/audit.log (I think that is the path) to the stage dir there. Then depends on that from your other changes | 16:43 |
| kashyap | clarkb: Ah, so a Do-Not-Merge patch to do that would be a quick way to test? | 16:44 |
| clarkb | kashyap: ya though we may want to merge that and collect that data going forward if it would be useful | 16:45 |
| kashyap | clarkb: My patch is merged in master (it's the version bump of libvirt/QEMU and related compat code clean-up) | 16:45 |
| kashyap | clarkb: So something like: | 16:47 |
| kashyap | if [ `command -v dpkg` ]; then | 16:47 |
| kashyap | /var/log/audit.log |& tee {{ stage_dir }}/audit.txt | 16:47 |
| *** mattw4 has joined #openstack-infra | 16:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run zuul-preview https://review.openstack.org/637654 | 16:47 |
| clarkb | kashyap: or more generally if [ -f /var/log/audit.log ] ; then sudo cp /var/log/audit.log {{ stage_dir }}/audit.log && chmod +r {{ stage_dir }}/audit.log ; fi ? | 16:48 |
| kashyap | Ah | 16:49 |
| kashyap | clarkb: Yeah, that's nicer, too | 16:49 |
| kashyap | clarkb: Shall I send a patch with 'Suggested-by' to you? | 16:50 |
| kashyap | Unless you've already published one :-) | 16:50 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add docker image jobs https://review.openstack.org/638707 | 16:50 |
| clarkb | kashyap: push it however you like :) | 16:50 |
| kashyap | I'll send one with attribution | 16:50 |
| *** diablo_rojo has joined #openstack-infra | 16:52 | |
| *** rossella_s has joined #openstack-infra | 16:53 | |
| *** roman_g has quit IRC | 16:54 | |
| *** agopi is now known as agopi|lunch|trav | 16:54 | |
| *** agopi|lunch|trav has quit IRC | 16:55 | |
| *** evrardjp is now known as gatersaregonnaga | 16:58 | |
| *** ginopc has quit IRC | 16:59 | |
| *** gatersaregonnaga is now known as evrardjp | 17:00 | |
| kashyap | clarkb: Is there an audit.log at all on Ubuntu? I was told /var/log/auth.log is it | 17:01 |
| clarkb | kashyap: let me check my local server | 17:01 |
| kashyap | I guess it's /var/log/audit, if the audit package is installed | 17:01 |
| kashyap | http://manpages.ubuntu.com/manpages/trusty/man5/auditd.conf.5.html | 17:01 |
| clarkb | my opensuse + apparmor install has /var/log/audit/audit.log. My ubuntu xenial install does not | 17:02 |
| clarkb | https://wiki.ubuntu.com/DebuggingApparmor says it is in the kern.log which we do already log | 17:03 |
| kashyap | Okido, thanks | 17:04 |
| kashyap | Ah | 17:04 |
| clarkb | kashyap: it is included ins syslog log file iirc | 17:04 |
| kashyap | kern.log? Where is it even; /me goes to look in | 17:04 |
| kashyap | http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/ | 17:04 |
| clarkb | we do a journalctl -u kernel -u otherstuff or similar for that file | 17:04 |
| kashyap | Ah-ha | 17:05 |
| clarkb | https://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/export-devstack-journal/tasks/main.yaml#n29 yup | 17:05 |
| clarkb | -t kernel | 17:05 |
| kashyap | So it should be it: http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/syslog.txt.gz | 17:06 |
| kashyap | (In my case) | 17:06 |
| clarkb | yes I think so | 17:06 |
| *** roman_g has joined #openstack-infra | 17:06 | |
| *** mattw4 has quit IRC | 17:07 | |
| kashyap | clarkb: So ... there are no AppArmor entries there. Talking to Sean Mooney, they said it should be 'dmesg' | 17:07 |
| *** ykarel_ has joined #openstack-infra | 17:08 | |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| kashyap | clarkb: Ha! Found the sucker ... in my problem scenario: | 17:08 |
| *** mattw4 has joined #openstack-infra | 17:09 | |
| kashyap | Feb 21 17:14:13 ubuntu-bionic-inap-mtl01-0002851272 kernel: traps: qemu-system-x86[31240] general protection ip:5600cbedaf78 sp:7f2dba1ebf00 error:0 in qemu-system-x86_64[5600cb81c000+8d2000] | 17:09 |
| kashyap | 'the hell is that... | 17:09 |
| *** ykarel|away has quit IRC | 17:10 | |
| kashyap | clarkb: Do you know what that 'trap' is trying to tell us? Or who actually knows AppArmor? | 17:10 |
| *** helenaAM has quit IRC | 17:11 | |
| openstackgerrit | Merged openstack-infra/puppet-kerberos master: Workaround broken ubuntu packaging https://review.openstack.org/638570 | 17:13 |
| clarkb | kashyap: is that the kernel trapping a general protection fault from the cpu? | 17:16 |
| kashyap | It's actually a good old QEMU crash | 17:16 |
| kashyap | That's what at least a seasoned QEMU dev said | 17:16 |
| clarkb | ya I think that is qemu doing a thing that causes a general protection fault | 17:17 |
| *** luizbag has joined #openstack-infra | 17:17 | |
| kashyap | clarkb: I have an evil question: what is the hack to upload (now that 'rootwrap' in Nova is gone) a debug QEMU binary build to get backtraces? | 17:17 |
| *** trown is now known as trown|lunch | 17:18 | |
| clarkb | kashyap: you'd want a devstack change that replaces qemu after the regular qemu install in devstack (fwiw this was probabl always a better way to do it. THe rootwrap hack was only a thing I think because no one wanted to read devstack bash) | 17:18 |
| *** rossella_s has quit IRC | 17:18 | |
| clarkb | kashyap: in devstack/lib/nova there should be a section of the code that configures libvirt and qemu. I would do your updates just befoer that | 17:18 |
| clarkb | and ubuntu probably even has packages for that? you might be able to just change the devstack/files/deb/nova list to be qemu-debug instead of qemu | 17:19 |
| clarkb | kashyap: looks like we'd have to modify the /etc/apt/sources.list to pull from ddebs.ubuntu.com then you should be able to install qemu-dbg ? | 17:22 |
| * kashyap will pay 100% attention here shortly; talking to QEMU dev | 17:23 | |
| clarkb | or do it however you were doing it before but have devstack do it for you | 17:23 |
| kashyap | I don't know how you folks do, splitting b/n multiple channels. Despite 10 years of IRC, I still feel like an "headless chicken" running around | 17:23 |
| kashyap | :D | 17:23 |
| *** yamamoto has joined #openstack-infra | 17:24 | |
| kashyap | clarkb: Haha ("no one wanted to read DevStack Bash") | 17:24 |
| kashyap | clarkb: Okay, so is there an existing patch that did via DevStack? (Doesn't have to be QEMU, another binary is fine, too) | 17:25 |
| clarkb | kashyap: I am not aware of one. But if we figure this out it might be a nice idea to add DEBUG_QEMU as a flag and toggle it on and off | 17:25 |
| kashyap | Oh, yeah... | 17:25 |
| kashyap | Good idea | 17:25 |
| *** markvoelker has joined #openstack-infra | 17:25 | |
| corvus | clarkb, mordred, fungi: i'm doing a final cleanup pass over the registry jobs. i'll try to batch my review requests. the first batch is ready -- the 4 changes with V+1 votes in https://review.openstack.org/#/q/status:open+topic:registry are ready to merge | 17:26 |
| corvus | clarkb, mordred, fungi: (at the end of this, we should have a zuul-preview server running) | 17:27 |
| clarkb | kashyap: https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n289 is probable the easiest place to add something for now | 17:27 |
| * kashyap clicks | 17:27 | |
| *** tosky has quit IRC | 17:28 | |
| *** yamamoto has quit IRC | 17:29 | |
| kashyap | clarkb: I certainly would find it really useful; even for libvirt binary | 17:30 |
| kashyap | DEBUG-LIBVIRTD or something like that | 17:30 |
| kashyap | s/-/_ | 17:30 |
| *** luizbag_ has joined #openstack-infra | 17:32 | |
| clarkb | corvus: the reason the docker image promotion pipeline is in opendev base jobs and not zuul jobs is that we need them to be privileged right? | 17:33 |
| *** luizbag has quit IRC | 17:34 | |
| clarkb | (fairly certain that is the case as they rely on that registry secret) | 17:35 |
| *** jpich has quit IRC | 17:36 | |
| corvus | clarkb: right, the pre and post run playbooks run privileged roles on the executor... i think since they execute skopeo they need to be in a trusted playbook. | 17:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove buildset registry job https://review.openstack.org/638706 | 17:38 |
| clarkb | corvus: posted a different question on https://review.openstack.org/#/c/638707/2 | 17:39 |
| corvus | clarkb: replied | 17:44 |
| clarkb | corvus: but we don't provide docker_credentials to those jobs? | 17:45 |
| clarkb | corvus: or are you saying you must inherit from that job and supply those creds yourself? | 17:45 |
| corvus | clarkb: the link i left there points to a change which does | 17:45 |
| corvus | clarkb: yep. | 17:45 |
| clarkb | corvus: if those jobs are inheritable couldn't you inherit from them and echo the credentials for the intermediate (and buildset registries) | 17:46 |
| kashyap | clarkb: Thanks for the (non-null) pointers :-) Appreciate it | 17:46 |
| clarkb | I guess that would depend on when those credentials are written to disk | 17:46 |
| clarkb | kashyap: no problem | 17:46 |
| *** takamatsu_ has joined #openstack-infra | 17:48 | |
| clarkb | it looks like we may have intermediate registry creds written in pre | 17:48 |
| clarkb | (because we pull from that registry in pre) | 17:48 |
| *** takamatsu has quit IRC | 17:48 | |
| corvus | clarkb: if you echo the buildset registry (which you can do, it's all there in an ansible variable), you're only shooting yourself in the foot, no harm done to the system. the intermediate registry credential is only available to the pre and post playbooks in that base job. the creds are never written to disk, only used on a command line which runs on the executor. that secret isn't available | 17:49 |
| corvus | to any other playbooks. | 17:49 |
| corvus | the roles in pre and post which use the intermediate registry secret will be (though they are not right now) no_log, so that it won't be exposed via ansible | 17:49 |
| clarkb | I see it is only the buildset registry that we write creds to disk for | 17:49 |
| corvus | correct | 17:50 |
| clarkb | and ya those creds are not really a problem since it is an ephemeral registry | 17:50 |
| corvus | after all this lands and is confirmed to work, i'm going to enable no_log on those and change our intermediate registry password (beacuse i have intentionally exposed it during debugging) | 17:50 |
| corvus | correct | 17:50 |
| clarkb | ok approved the change. Remainign feedback is maybe add a blurb about inheriting from those jobs with your specific credentials for dockerhub publishing (I couldn't find that in the existing rst files) | 17:52 |
| *** sdake has quit IRC | 17:54 | |
| corvus | well, the rst file describes the input for the job just like all the other jobs; inheritance is always optional. | 17:55 |
| *** sdake has joined #openstack-infra | 17:55 | |
| clarkb | you won't be able to do the promote job without inhertance right? | 17:55 |
| clarkb | (and the publish job) | 17:55 |
| *** betherly has joined #openstack-infra | 17:56 | |
| *** ijw has joined #openstack-infra | 17:57 | |
| corvus | you just need to give the job your secret. you should probably do that via inheritance, but you could just invoke "opendev-promote-docker-image" in a project-pipeline and give it the secret there. | 17:57 |
| corvus | if we set "abstract: true" we could force people to make new jobs and inherit. | 17:57 |
| clarkb | I see | 17:57 |
| *** roman_g has quit IRC | 17:58 | |
| *** electrofelix has quit IRC | 17:58 | |
| *** markvoelker has quit IRC | 17:59 | |
| *** raissa has quit IRC | 17:59 | |
| *** betherly has quit IRC | 18:01 | |
| clarkb | ok puppet worked on kdc04 and stopped kadmind and set it to disabled. krb5-kdc is not running but systemctl seems to imply it is enabled. I'm going to reboot that server to see if it comes up as we want with no kadmind and a krb5-kdc running | 18:01 |
| *** wolverineav has joined #openstack-infra | 18:01 | |
| openstackgerrit | Merged opendev/base-jobs master: Add docker image jobs https://review.openstack.org/638707 | 18:02 |
| *** takamatsu_ has quit IRC | 18:03 | |
| clarkb | it came up as expected. i think kdc04 is in a happy spot now | 18:03 |
| *** mriedem is now known as mriedem_lunch | 18:05 | |
| *** takamatsu_ has joined #openstack-infra | 18:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test https://review.openstack.org/638736 | 18:07 |
| *** jpena is now known as jpena|off | 18:07 | |
| *** ekultails has quit IRC | 18:12 | |
| *** ekultails has joined #openstack-infra | 18:13 | |
| *** jmorgan1 has quit IRC | 18:15 | |
| corvus | clarkb: i just double checked -- there is something that requires it to be a trusted playbook: http://logs.openstack.org/36/638736/1/check/corvus-test/59c5003/ara-report/result/8f9ec055-fc73-4116-a496-02caf54e1675/ | 18:15 |
| *** dave-mccowan has quit IRC | 18:15 | |
| corvus | clarkb: however, that may be the only thing | 18:15 |
| corvus | unfortunately, i think skopeo is hard-coded to look in /etc/docker/certs.d (and a few other /etc/ locations). so we can't just make a workdir/certs.d | 18:17 |
| clarkb | darn | 18:17 |
| *** nicolasbock has joined #openstack-infra | 18:17 | |
| clarkb | I am booting kdc03.openstack.org now (as mentioned before I think opendev kerberos servers are currently out of scope of current work as that would imply completely new realm setup) | 18:17 |
| clarkb | once that is up I'll get a change up to system-config to add it to site.pp and the inventory and all that | 18:18 |
| corvus | clarkb: ++ | 18:18 |
| nicolasbock | Hi, where is `NODEPOOL_MIRROR_HOST` defined? | 18:18 |
| nicolasbock | I found multiple references to it, but not where it's defined | 18:18 |
| clarkb | nicolasbock: it is set in a legacy compatibility script at /etc/nodepool/mirror_info.sh that you can source. Let me find where we define that | 18:19 |
| nicolasbock | Oh cool | 18:19 |
| clarkb | nicolasbock: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/mirror-info/templates/mirror_info.sh.j2#n17 | 18:20 |
| nicolasbock | The reason I was asking is that we are seeing RPM repository verification errors (http://logs.openstack.org/47/638547/1/gate/openstack-ansible-functional-opensuse-423/7ec16b6/job-output.txt.gz#_2019-02-22_15_04_59_341959) that could be caused by an outdated mirror | 18:20 |
| corvus | http://git.openstack.org/cgit/opendev/base-jobs/tree/roles/mirror-info/templates/mirror_info.sh.j2 | 18:20 |
| clarkb | nicolasbock: we've had that problem in the past due to opensuse rsync servers rejecting our connections | 18:20 |
| nicolasbock | This mirror is mariadb | 18:21 |
| nicolasbock | They are building openSUSE packages | 18:21 |
| *** jmorgan1 has joined #openstack-infra | 18:21 | |
| clarkb | http://mirror.ord.rax.openstack.org:8080/MariaDB/mariadb-10.2.17/yum/opensuse42-amd64 specifically is what is failing and that isn't a mirror but a cache | 18:21 |
| clarkb | er caching proxy | 18:21 |
| *** jmorgan1 has quit IRC | 18:21 | |
| nicolasbock | Ah | 18:22 |
| clarkb | you should be able to navigate it and check things directly | 18:22 |
| *** jmorgan1 has joined #openstack-infra | 18:22 | |
| *** jamesmcarthur has quit IRC | 18:22 | |
| nicolasbock | You mean go through the proxy URI? | 18:22 |
| nicolasbock | I'll try to manually verify that repo | 18:22 |
| clarkb | nicolasbock: ya open that link above | 18:22 |
| nicolasbock | Thanks for the pointer! | 18:22 |
| *** whoami-rajat has quit IRC | 18:27 | |
| *** rlandy is now known as rlandy|brb | 18:28 | |
| *** wolverineav has quit IRC | 18:28 | |
| *** wolverineav has joined #openstack-infra | 18:31 | |
| corvus | clarkb: i stand corrected, there is a skopeo option to set the cert dir | 18:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add kdc03.openstack.org https://review.openstack.org/638745 | 18:32 |
| clarkb | corvus: yay | 18:32 |
| corvus | clarkb: sorry for the churn, but i think we might be able to move this into zuul-jobs after all. :) | 18:32 |
| *** wolverineav has quit IRC | 18:32 | |
| *** wolverineav has joined #openstack-infra | 18:32 | |
| corvus | i'm going to take a little break and then work up those changes | 18:32 |
| clarkb | infra-root I think ^ is ready to be approved. I'm going to add dns records now (including the SRV record since kdc04 had no kdc running and afs was happy we should be fine for a bit while we wait for puppet to configure kdc03). I'll also add the host principle (and maybe even figure out how to remove the kdc02 principle) | 18:34 |
| *** gfidente has quit IRC | 18:34 | |
| clarkb | then once puppet is done on kdc03 I'll copy the stash file making it an eligible master node. And we can plan for a swap of the master nodes next week | 18:35 |
| *** betherly has joined #openstack-infra | 18:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Build docker image https://review.openstack.org/637037 | 18:39 |
| *** ramishra has quit IRC | 18:40 | |
| *** trown|lunch is now known as trown | 18:40 | |
| *** betherly has quit IRC | 18:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run zuul-preview https://review.openstack.org/637654 | 18:40 |
| clarkb | while I'm thinking about this kerberos stuff ansible likely would model the interaction with kerberos itself and the need for copying files between nodes. I will follow our docs as is for now but if/when we do an opendev.org realm we ought to be able to use playbooks to add principles, copy the keytab, then copy the stash | 18:40 |
| *** chandankumar is now known as raukadah | 18:45 | |
| *** agopi has joined #openstack-infra | 18:45 | |
| *** jamesmcarthur has joined #openstack-infra | 18:51 | |
| *** lpetrut has quit IRC | 18:51 | |
| *** rlandy|brb is now known as rlandy | 18:54 | |
| *** markvoelker has joined #openstack-infra | 18:56 | |
| *** bhavikdbavishi has quit IRC | 18:57 | |
| clarkb | infra-root I've added all the dns records for kdc03.o.o including the SRV record. I've also updated the krb5.keytab on all three kdcs and kprop from 01 to 04 continues to work. We are ready for https://review.openstack.org/638745 (thank you corvus for the review) | 19:00 |
| nicolasbock | clarkb: I tested the proxy URI from above and it works without any errors. Could the proxy cache have been out of date when the gate job ran? | 19:02 |
| clarkb | nicolasbock: that could be. Perhaps the metadata and the actual contents were cached differently (so one had updated while the other was not yet expired) | 19:03 |
| nicolasbock | Ok, is that something we can do anything about? | 19:04 |
| nicolasbock | What's strange though is that this version is not the latest and I wouldn't expect any changes in that repo | 19:04 |
| clarkb | nicolasbock: you could check the headers on the files when requested from the backend to ensure they aren't wildly different expiry times | 19:04 |
| clarkb | and if they are wildly different ask the backend to fix it to be more friendly to a cache | 19:04 |
| clarkb | (another option is to not use the cache and talk directly to the backend though that tends to fail for other reasons) | 19:04 |
| nicolasbock | Ok, thanks! | 19:05 |
| clarkb | fungi: I'm trying to update my trusty server upgrade todo list. Are storyboard and storyboard-dev as well as openstackid and openstackid-dev all upgraded to xenial now? I'll mark them done if so | 19:06 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Assure iptables is installed inside multi-node-firewall role https://review.openstack.org/638414 | 19:06 |
| *** lpetrut has joined #openstack-infra | 19:07 | |
| clarkb | cmorpheus: not sure if you saw but I got afs and kerberos servers futureparsered | 19:08 |
| clarkb | fungi: smarcet, if the openstackid servers are done can we move forward with https://review.openstack.org/#/c/616001/8 and its child now? | 19:09 |
| clarkb | I'm happy to help with ^ but want ot make sure that won't interfere with your work | 19:09 |
| *** yamamoto has joined #openstack-infra | 19:12 | |
| *** jamesmcarthur has quit IRC | 19:12 | |
| *** auristor has quit IRC | 19:15 | |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| *** yamamoto has quit IRC | 19:16 | |
| *** e0ne has joined #openstack-infra | 19:18 | |
| *** auristor has joined #openstack-infra | 19:22 | |
| *** mriedem_lunch is now known as mriedem | 19:25 | |
| *** eernst has joined #openstack-infra | 19:26 | |
| *** ykarel_ is now known as ykarel | 19:28 | |
| *** markvoelker has quit IRC | 19:28 | |
| *** eernst has quit IRC | 19:31 | |
| *** sdake has quit IRC | 19:31 | |
| corvus | clarkb: i think i'm going to defer the move to zuul-jobs -- we'd need the same number of jobs in opendev/base-jobs, we'd just move some simple playbooks out. since this is in base-jobs anyway (versus leaf-node jobs in project repos), i'm not convinced it's worth much effort atm. we can always do it later. | 19:32 |
| *** ykarel is now known as ykarel|away | 19:32 | |
| clarkb | corvus: wfm | 19:32 |
| *** eernst has joined #openstack-infra | 19:34 | |
| cloudnull | hey all, ive not been following along but we're seeing a bunch of these errors "... finger://ze03.openstack.org/5a2c579048f64e80925ba22aafd6c0b0 : RETRY_LIMIT" - is that something known? | 19:36 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Remove pre playbook from opendev-build-docker-image https://review.openstack.org/638758 | 19:36 |
| clarkb | cloudnull: no, what job was that for? | 19:36 |
| corvus | clarkb: ^ that's an oops and needs to go in before the rest will work | 19:36 |
| cloudnull | clarkb https://review.openstack.org/#/c/638556/ | 19:37 |
| corvus | AJaeger: https://review.openstack.org/638758 could use a quick review if you have a sec | 19:37 |
| *** eernst has quit IRC | 19:38 | |
| corvus | clarkb, clarkb: 2019-02-22 16:18:56,647 DEBUG zuul.AnsibleJob: [build: 5a2c579048f64e80925ba22aafd6c0b0] Ansible complete, result RESULT_UNREACHABLE code None | 19:39 |
| AJaeger | corvus: just done - was double checking your comment ;) | 19:39 |
| clarkb | cloudnull: http://paste.openstack.org/show/745765/ looks like the job may have killed the node? or the node went unreachable due to duplication ip problem? | 19:40 |
| *** eernst has joined #openstack-infra | 19:40 | |
| cloudnull | could be? | 19:41 |
| cloudnull | if you check that job out we've rechecked it a bunch, its always finger://ze*.openstack.org... | 19:42 |
| clarkb | cloudnull: if you watch the log stream while the job is running that may give you more info | 19:42 |
| cloudnull | ok. | 19:42 |
| cloudnull | will have a look | 19:42 |
| *** dave-mccowan has joined #openstack-infra | 19:43 | |
| *** eernst has quit IRC | 19:44 | |
| *** eernst has joined #openstack-infra | 19:46 | |
| *** ykarel|away has quit IRC | 19:46 | |
| *** eernst has quit IRC | 19:48 | |
| corvus | clarkb, cloudnull: i spot checked 5 of those, they ran in ovh-bhs1 x2, rax-iad x1, and rax-ord x2 | 19:50 |
| corvus | that diversity of providers makes an infrastructure problem (such as IP reuse) unlikely i think. | 19:50 |
| clarkb | ++ | 19:50 |
| corvus | cloudnull: also not sure if you noticed, but this job failed, but managed to finish and upload logs. which means it could be a different problem, or it could be related to the same problem and might help you triangulate it: http://logs.openstack.org/56/638556/1/check/openstack-ansible-functional-distro_install-centos-7/2541dda/ | 19:51 |
| fungi | okay, taxes and lunch are behind me... now just have some hundreds of lines of scrollback in here to catch up on | 19:51 |
| corvus | fungi: every line is precious and will be a joy for you to read | 19:52 |
| clarkb | fungi: start here https://review.openstack.org/#/c/638745/1 :) | 19:52 |
| *** yamamoto has joined #openstack-infra | 19:52 | |
| *** Vadmacs has quit IRC | 19:52 | |
| *** luizbag_ has quit IRC | 19:53 | |
| cmorpheus | clarkb: ya i saw :D i rebased the other puppet4 switches | 19:53 |
| *** yamamoto has quit IRC | 19:57 | |
| *** wolverineav has quit IRC | 19:59 | |
| *** eernst has joined #openstack-infra | 19:59 | |
| *** wolverineav has joined #openstack-infra | 19:59 | |
| *** eernst has quit IRC | 20:01 | |
| fungi | clarkb: yes, the storyboard and openstackid servers are all running on xenial now. there may still be a bit of cleanup to do but it should be safe to check them off the list | 20:03 |
| *** wolverineav has quit IRC | 20:04 | |
| clarkb | fungi: great. I had a followup question on whether or not we could merge the futureparser updates for openstackid servers. If you could review those (assuming we are ready) I can approve and babysit | 20:04 |
| *** wolverineav has joined #openstack-infra | 20:04 | |
| clarkb | the list of trusty servers gets shorter and shorter :) hopefully I'lld have kdc01 gone sometime monday | 20:05 |
| *** jcoufal has quit IRC | 20:05 | |
| fungi | yeah the puppet 4 change should be fine but we'll want to keep an eye on syslog for openstackid.org definitely | 20:06 |
| *** lpetrut has quit IRC | 20:08 | |
| clarkb | ya we can do -dev first and make sure it looks happy befor edoing prod | 20:08 |
| *** e0ne has quit IRC | 20:08 | |
| fungi | that should be a proper test at this point now that they're both back in sync | 20:09 |
| *** wolverineav has quit IRC | 20:09 | |
| openstackgerrit | Merged opendev/base-jobs master: Remove pre playbook from opendev-build-docker-image https://review.openstack.org/638758 | 20:11 |
| clarkb | ya this was one of the reasons for waiting | 20:12 |
| *** ekultails has quit IRC | 20:25 | |
| *** markvoelker has joined #openstack-infra | 20:25 | |
| fungi | what's our stance on extending devstack-gate these days? came up in the context of tbarron's https://review.openstack.org/626921 | 20:26 |
| openstackgerrit | Merged openstack-infra/system-config master: Don't install a blank docker daemon config https://review.openstack.org/638199 | 20:26 |
| corvus | fungi: that should totally be a zuulv3 job that inherits from devstack or tempest and adds a one-line change to the logging variable | 20:28 |
| tbarron | fungi: corvus: so iiuc the corresponding logging captures there for, say ceph or gluster, were pre-zuul-v3 | 20:29 |
| corvus | i'm not going to -1 something like that, but i don't think i should be +2ing it either. | 20:29 |
| tbarron | fungi: corvus: i do want to convert all our legacy jobs over, we just have reduced cores/participants and haven't gotten to that for this one yet | 20:29 |
| * fungi notes devstack-gate isn't exactly bursting with new core reviewers either ;) | 20:30 | |
| tbarron | fungi: yeah ..., sorry to add work :) | 20:30 |
| *** ijw has quit IRC | 20:31 | |
| openstackgerrit | Merged openstack-infra/system-config master: run-base: configure docker mirrors on all hosts in CI https://review.openstack.org/638200 | 20:31 |
| fungi | well, we did a bunch of extra work to create a new system which would let the individual teams take care of such things themselves so we could stop reviewing changes to unnecessarily centralized tools like devstack-gate | 20:31 |
| tbarron | fungi: corvus: naively, I didn't realize that I was proposing a mod to stuff that is only on the legacy path | 20:31 |
| tbarron | fungi: corvus: so i understand if you guys -2 it | 20:32 |
| *** ijw has joined #openstack-infra | 20:32 | |
| tbarron | fungi: corvus: i'll try to get that job converted over and can do a debug depends-on to this patch in the mean time when I need it | 20:32 |
| fungi | it's understandable. i think we didn't make a lot of noise about devstack-gate being cruft. it's more that the new-style devstack jobs inherit from playbooks and job definitions in the devstack repo where the qa team can take on responsibility for them alongside devstack itself | 20:33 |
| corvus | yeah, i don't mind if the change sits there for your use in debugging in the mean time :) | 20:33 |
| fungi | so devstack-gate is no longer in use (theoretically) by jobs other than those which were semi-automatically converted from the old zuul v2 configuration data | 20:34 |
| tbarron | fungi: corvus: my challenge is that I inherited legacy jobs with post-test-hook weirdness for our plugin and converting them w/o breaking everything is not straightforward | 20:34 |
| openstackgerrit | Merged openstack-infra/system-config master: Add kdc03.openstack.org https://review.openstack.org/638745 | 20:34 |
| tbarron | fungi: corvus: i do like the new system much better | 20:34 |
| corvus | i was about to say something about how it shouldn't be hard to port the jobs over, but boy am i glad i didn't. i would have put my foot squarely in my mouth. | 20:34 |
| tbarron | corvus: well it might not be once I understand the weird auto-converted-from-jjb-jobs that we have now | 20:35 |
| corvus | tbarron: if you have questions about how to accomplish something, i'm happy to help :) | 20:35 |
| tbarron | corvus: if I had grown up with the jjb ones it might be clearer | 20:35 |
| tbarron | corvus: thanks | 20:35 |
| fungi | i'll be the first to acknowledge there were plenty of funky devstack-gate hooks which grew into random projects over the years which we didn't necessarily provide a clear migration path to deal with | 20:36 |
| tbarron | "so devstack-gate is no longer in use (theoretically) by jobs other than those which were semi-automatically converted from the old zuul v2 configuration data" | 20:36 |
| clarkb | fwiw if you run that service as a devstack screen service it will be auto logged for you | 20:36 |
| tbarron | that's the part I didn't know, but could admittedly have inferred from looking at the new jobs | 20:36 |
| clarkb | this is true of legacy and modern devstack jobs | 20:37 |
| * corvus does some quick math to figure out how old someone would be who *literally* grew up with jjb. | 20:37 | |
| *** ijw has quit IRC | 20:37 | |
| tbarron | heh, I would qualify, but we weren't from the same neighborhood | 20:37 |
| fungi | i used to sprinkle it on my breakfast cereal as a kid | 20:38 |
| clarkb | corvus: older than my kids | 20:38 |
| clarkb | mid 2012ish iirc so almost 7 | 20:39 |
| tbarron | oh it's a lot newer than i realized | 20:39 |
| corvus | clarkb: yeah, but i think you can add another 4-6 years to that and still consider them "having grown up with" | 20:40 |
| corvus | depending on how precocious a kid is | 20:40 |
| clarkb | related. I was cleaning out my closet the other night and found my old jenkins bobblehead | 20:41 |
| clarkb | so that is sitting on my desk again | 20:41 |
| *** ijw has joined #openstack-infra | 20:42 | |
| *** ekultails has joined #openstack-infra | 20:43 | |
| *** wolverineav has joined #openstack-infra | 20:45 | |
| *** ijw has quit IRC | 20:47 | |
| *** wolverineav has quit IRC | 20:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 20:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid-dev https://review.openstack.org/616001 | 20:55 |
| cmorpheus | fungi: ^ | 20:55 |
| clarkb | cmorpheus: oh I was just looking at that rebase :) | 20:55 |
| clarkb | cmorpheus: it needs to be dev*.openstack.org or dev[0-9]*.openstack.org | 20:56 |
| cmorpheus | oh | 20:56 |
| clarkb | I would do dev[0-9]* | 20:56 |
| clarkb | the old non digited server should be deleted soon if not already | 20:56 |
| *** wolverineav has joined #openstack-infra | 20:56 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid-dev https://review.openstack.org/616001 | 20:57 |
| corvus | clarkb, fungi: while i'm waiting for some registry stuff, can you review https://review.openstack.org/637334 ? with that i can spin up some gitea servers. | 20:58 |
| *** ijw has joined #openstack-infra | 20:58 | |
| fungi | sure! | 20:58 |
| fungi | clarkb: good catch on the glob | 20:58 |
| *** markvoelker has quit IRC | 20:59 | |
| clarkb | fungi: looks like dns for prod openstackid still points at the old server? | 21:00 |
| fungi | does it? checking | 21:01 |
| fungi | no, that's the new server. compare the address records to openstackid01.openstack.org | 21:02 |
| openstackgerrit | Merged openstack-infra/nodepool master: doc bugfix for static provider https://review.openstack.org/637518 | 21:02 |
| clarkb | oh I see, no cname | 21:02 |
| fungi | they should match (they do for me) | 21:02 |
| clarkb | was that intentional? | 21:02 |
| clarkb | oh right | 21:02 |
| fungi | yes, because dns | 21:02 |
| clarkb | because its root | 21:02 |
| clarkb | ya | 21:02 |
| fungi | if we ran the org tld we could make a cname for openstackid in it | 21:03 |
| fungi | but alas we don't | 21:03 |
| clarkb | probably a good thing | 21:03 |
| fungi | just as well, yeah. i have enough on my plate anyway | 21:03 |
| *** ijw has quit IRC | 21:03 | |
| *** yamamoto has joined #openstack-infra | 21:04 | |
| clarkb | fwiw I didn't end up removing the kdc02 principal from the host keytab file. Decided to do that when kdc01 is completely gone and remove both at the same time | 21:05 |
| corvus | fungi: you might be interested in seeing the vulnerability information on this page: https://hub.docker.com/_/debian/?tab=tags | 21:06 |
| corvus | i have no opinions about it. i've never seen it before. i just happened to notice it. | 21:07 |
| openstackgerrit | Merged openstack-infra/nodepool master: Use a pipeline for dib stats https://review.openstack.org/638265 | 21:07 |
| fungi | where's the vulnerability information on that page? i'm likely going blind | 21:08 |
| *** yamamoto has quit IRC | 21:09 | |
| corvus | fungi: oh, wow, apparently that's something i see when i'm logged in | 21:09 |
| corvus | no wonder i haven't seen it before | 21:09 |
| corvus | and what kind of sense does that make? | 21:09 |
| fungi | about as much sense as the open-source security summit google just ran this week which was by invitation only | 21:09 |
| corvus | (how is vulnerability information user-contextual?) | 21:09 |
| corvus | only google knows whether you're a white or black hat. regardless of whether you even know. | 21:10 |
| *** tosky has joined #openstack-infra | 21:11 | |
| corvus | fungi: https://screenshots.firefox.com/ICZBtKxIEIsokoFt/hub.docker.com | 21:12 |
| fungi | definitely doesn't seem to jive with https://security-tracker.debian.org/tracker/CVE-2018-17953 | 21:13 |
| fungi | 1.1.3 is... ancient | 21:14 |
| *** ijw has joined #openstack-infra | 21:14 | |
| corvus | well, that was probably wheezy | 21:15 |
| corvus | probably would have been good of docker to include the tag name on that page | 21:15 |
| fungi | yeah, and identify images of distros which are long past their eol | 21:16 |
| corvus | but of course, it's docker. this is probably indexed by layer sha, and it's already forgot the tag. i can't believe you didint' know that 6d0abf8ba24e5ff4bfd111e705a6d33a547fca1f0751c22c5205c504a569a is wheezy. | 21:16 |
| corvus | oh, wait, that's unstable. | 21:16 |
| fungi | hah | 21:16 |
| fungi | i also can't find any record of a linux-pam binary package ever existing in debian itself | 21:17 |
| fungi | though here's the snapshot for the pam 1.1.3-7.1 source package http://snapshot.debian.org/package/pam/1.1.3-7.1/ | 21:18 |
| fungi | package changelog says that was uploaded in 2012 | 21:20 |
| *** ijw has quit IRC | 21:21 | |
| fungi | if the current debian/sid docker image has a 7-year-old pam build in it, i'll be surprised | 21:21 |
| fungi | as we just covered, that's as old as jjb, and older than clarkb's offspring | 21:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: run-buildset-container: fix username/password for proxy registry https://review.openstack.org/638767 | 21:23 |
| corvus | clarkb, fungi, AJaeger: ^ another oops | 21:23 |
| corvus | i have manually verified that fix works, it just didn't make it into my patch yesterday. happily, the current work is actually exercising that code path and caught it. | 21:24 |
| *** jcoufal has joined #openstack-infra | 21:24 | |
| clarkb | yay yaml I suppose | 21:24 |
| *** jcoufal has quit IRC | 21:25 | |
| fungi | that's a rather amusing result of layering parsers, i suppose | 21:25 |
| corvus | this is an interesting difference with docker-compose too, where env vars are treated as strings, not dicts. | 21:25 |
| *** wolverineav has quit IRC | 21:26 | |
| corvus | (so an earlier incarnation i had used "REGISTRY_PROXY_USERNAME=" which was fine in docker compose. | 21:26 |
| *** wolverineav has joined #openstack-infra | 21:27 | |
| *** kjackal has quit IRC | 21:28 | |
| fungi | mildly reminiscent of quiet nans vs signaling nans in the ieee 754 fp standard. we need a quiet propagating null value which is only rendered to an actual null by the uppermost parser layer | 21:31 |
| *** betherly has joined #openstack-infra | 21:32 | |
| *** wolverineav has quit IRC | 21:37 | |
| *** betherly has quit IRC | 21:37 | |
| *** yamamoto has joined #openstack-infra | 21:39 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use host networking for gitea https://review.openstack.org/637334 | 21:42 |
| openstackgerrit | Eric Harney proposed openstack-infra/elastic-recheck master: Add query for tempest bug 1812036 https://review.openstack.org/638771 | 21:42 |
| openstack | bug 1812036 in tempest "Tests creating encrypted volume types can conflict" [Undecided,New] https://launchpad.net/bugs/1812036 | 21:43 |
| *** yamamoto has quit IRC | 21:43 | |
| corvus | clarkb, fungi: looks like gitea haproxy is ready now too: https://review.openstack.org/638033 | 21:45 |
| clarkb | Database propagation to kdc03.openstack.org: SUCCEEDED | 21:46 |
| clarkb | but krb5-kdc doesn't seem to be running so I am going to reboot like I did on 04 | 21:46 |
| * clarkb updates packages first | 21:47 | |
| tonyb | Now that https://review.openstack.org/#/c/638527/ has merged (thanks!) do I need to wait for that to be applied or are the vhosts regenerated automagically after merge? | 21:48 |
| *** ijw has joined #openstack-infra | 21:49 | |
| clarkb | tonyb: you'll need to wait for a puppet pulse, but that should've been done a long time ago | 21:50 |
| tonyb | clarkb: Okay thanks. Any chance I can request one next week? (when y'all will be around to watch it?) | 21:52 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: run-buildset-container: fix username/password for proxy registry https://review.openstack.org/638767 | 21:52 |
| *** ijw has quit IRC | 21:54 | |
| *** ijw has joined #openstack-infra | 21:55 | |
| *** dave-mccowan has quit IRC | 21:55 | |
| *** markvoelker has joined #openstack-infra | 21:56 | |
| clarkb | tonyb: Oh i mean it is done automagically. Are you seeing that things didn't apply as expected? | 21:57 |
| tonyb | clarkb: Oh my bad I misunderstood | 21:58 |
| tonyb | clarkb: I haven't checked ... | 21:59 |
| tonyb | clarkb: Yup it's working now | 21:59 |
| *** jtomasek has quit IRC | 21:59 | |
| tonyb | Thanks | 21:59 |
| clarkb | great | 21:59 |
| clarkb | thinks work the way I expect them to then :) | 21:59 |
| tonyb | \o/ | 21:59 |
| tonyb | clarkb: I love it when that happens | 22:00 |
| *** ijw has quit IRC | 22:00 | |
| diablo_rojo | tonyb, shouldn't you be enjoying your weekend? ;) | 22:00 |
| tonyb | diablo_rojo: I probably should be | 22:01 |
| tonyb | diablo_rojo: I ducked in to check on some work and thought I'd check for review commenst and saw that $stuff has merged | 22:01 |
| tonyb | diablo_rojo: so then I got all excied and started with the next step | 22:02 |
| diablo_rojo | tonyb, I can't judge, I was up till like...23:15 answering questions in #openstack-dev for a potential outreachy intern | 22:02 |
| diablo_rojo | Just teasing :) | 22:02 |
| tonyb | diablo_rojo: All good. | 22:03 |
| *** rh-jelabarre has quit IRC | 22:03 | |
| diablo_rojo | Its good to be excited about what you're working on :) | 22:03 |
| tonyb | diablo_rojo: Yup! Makes working on weekends fun! | 22:05 |
| *** ijw has joined #openstack-infra | 22:05 | |
| diablo_rojo | tonyb, ha ha we are sick individuals | 22:06 |
| * tonyb nods sagely | 22:06 | |
| tonyb | ;P | 22:06 |
| * tonyb is going to go fix bikes for the kids | 22:07 | |
| tonyb | Enjoy what's left of your Friday and have a great weekend | 22:08 |
| diablo_rojo | tonyb, you too :) | 22:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for openstackid-dev https://review.openstack.org/616001 | 22:11 |
| fungi | now that's ^ merged i'm tailing the syslog to see what happens | 22:12 |
| clarkb | I get krb5-kdc running (which requried the stash file from the master fwiw. I'll update docs as soon as I fix the next thing) and then kpropd stops running | 22:12 |
| clarkb | so I'm close on kdc03 but not quite there yet | 22:12 |
| fungi | did we miss putting some of the secret material under configuration management? | 22:13 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add run playbook to opendev-buildset-registry https://review.openstack.org/638776 | 22:14 |
| corvus | fungi, clarkb: another oops ^ | 22:14 |
| *** rlandy has quit IRC | 22:15 | |
| clarkb | fungi: none of the secret material is under config management as far as I can tell | 22:15 |
| clarkb | I see the krb5-kpropd issue. I'll get a patch up for that | 22:16 |
| corvus | most of the secret material is in the form of one-off "create a principal" commands. | 22:16 |
| *** diablo_rojo has quit IRC | 22:18 | |
| fungi | ahh, so just needs to be created anew per kdc | 22:18 |
| prometheanfire | tonyb: thanks | 22:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-kerberos master: Simplify service management https://review.openstack.org/638777 | 22:20 |
| clarkb | fungi: yes. One exception is the ecrypted stash file which is consistent and could be puppeted. But I'll leave that as an exercise for later for now. And instead document that fact | 22:20 |
| clarkb | infra-root ^ should be the fix for running krb5-kpropd on boot | 22:20 |
| clarkb | I'll write the docs update as soon as I'm done with reviwe of corvus oops fix | 22:21 |
| clarkb | fungi: fwiw I think ansible is relatively well suited to the tasks needed to create and sync the secret data, but we'll need to write that management tooling | 22:22 |
| clarkb | fungi: things like run kadmin.local command on master node. Copy results to all nodes. | 22:23 |
| corvus | clarkb: ++ | 22:23 |
| fungi | and i guess 638776 is not going to run speculatively as it's in opendev/base-jobs | 22:23 |
| corvus | fungi: correct, https://review.openstack.org/637654 is broken waiting on that to land | 22:24 |
| fungi | that's what i thought. cool | 22:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Document kerberos stash file requirement https://review.openstack.org/638779 | 22:28 |
| clarkb | and now documented at least | 22:28 |
| corvus | clarkb, fungi: https://review.openstack.org/637037 is the penultimate change and is ready to land | 22:29 |
| *** markvoelker has quit IRC | 22:30 | |
| clarkb | I've learned so much about kerberos | 22:30 |
| clarkb | and debian packaging and debconf | 22:30 |
| clarkb | the dream of the 90s is a live in portland or is it openstack infra >_> | 22:30 |
| corvus | clarkb: for you, it's both :) | 22:30 |
| clarkb | indeed | 22:31 |
| corvus | though kerberos (unlike some other stuff from the 90s we run) is decidedly relevant. | 22:31 |
| fungi | even microsoft thinks so | 22:32 |
| openstackgerrit | Merged opendev/base-jobs master: Add run playbook to opendev-buildset-registry https://review.openstack.org/638776 | 22:32 |
| clarkb | ya I'm honestly quite surprised no one else has run into this packaging bug | 22:32 |
| clarkb | because kerberos is definitely still a thing | 22:32 |
| clarkb | which reminds me I need to figureo ut filing that bug | 22:32 |
| clarkb | I'm thinking a docker container might be the easiest way to reproduce on debian so that I don't end up being the person that says "please to fix in debian but I only tested on ubuntu" | 22:33 |
| fungi | i mean, you could open a bug in lp against the ubuntu package and let the maintainer forward it | 22:34 |
| clarkb | oh hrm I think the same person is actually listed in both places so maybe that woks well enough | 22:34 |
| fungi | though if you really want to reproduce and file it in debian i've got plenty of pointers | 22:35 |
| clarkb | no I think I've decided launchpad is easier :P | 22:35 |
| *** lefteri5 has joined #openstack-infra | 22:35 | |
| clarkb | the maintainer of the package seems to be the same in both places so I think this will be fine | 22:35 |
| fungi | yeah, i can see the allure ;) | 22:36 |
| *** sdake has joined #openstack-infra | 22:37 | |
| *** potsmaster has joined #openstack-infra | 22:40 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid https://review.openstack.org/616002 | 22:44 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for refstack https://review.openstack.org/628153 | 22:44 |
| clarkb | https://bugs.launchpad.net/ubuntu/+source/krb5/+bug/1817376 | 22:44 |
| openstack | Launchpad bug 1817376 in krb5 (Ubuntu) "krb5-admin-server postinst has broken debconf if RUN_KADMIND set to false in /etc/default/krb5-admin-server" [Undecided,New] | 22:44 |
| *** rkukura_ has joined #openstack-infra | 22:44 | |
| openstackgerrit | Merged openstack-infra/zuul-preview master: Build docker image https://review.openstack.org/637037 | 22:45 |
| *** rkukura has quit IRC | 22:47 | |
| *** rkukura_ is now known as rkukura | 22:47 | |
| corvus | w00t https://hub.docker.com/r/zuul/zuul-preview exists | 22:48 |
| clarkb | success | 22:49 |
| clarkb | cloudnull: any luck following the console log to see what was happening? | 22:49 |
| *** yamamoto has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 23:00 |
| *** yamamoto has quit IRC | 23:00 | |
| *** diablo_rojo has joined #openstack-infra | 23:05 | |
| clarkb | fungi: I think puppet runs on openstackid dev in the next 5 minutes or so | 23:10 |
| fungi | yeah, last run completed at 22:31 | 23:13 |
| rm_work | Is devstack supported in this channel or #openstack-dev? | 23:13 |
| *** rascasoft has quit IRC | 23:13 | |
| rm_work | or another option I didn't consider :P | 23:13 |
| fungi | #openstack0qa | 23:13 |
| rm_work | ahh k thx :) | 23:14 |
| fungi | er, #openstack-qa | 23:14 |
| fungi | the qa team maintains devstack | 23:14 |
| fungi | (and tempest, and grenade and a host of other test tools) | 23:14 |
| rm_work | cool, thanks | 23:17 |
| rm_work | I appreciate that you folks always know exactly who to talk to or who owns pretty much anything :) | 23:17 |
| rm_work | <3 | 23:17 |
| fungi | well, we have a cheat sheet | 23:17 |
| fungi | https://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml | 23:18 |
| fungi | and now, so do you! | 23:18 |
| corvus | hrm, launch-node failed to connect to ssh on the servers i just tried to launch in vexxhost-sjc1 | 23:21 |
| corvus | i used --network="public" and it was given a public ip | 23:22 |
| clarkb | security grops maybe? | 23:22 |
| corvus | clarkb: is that attached to the network? | 23:23 |
| clarkb | fungi: Feb 22 23:22:17 openstackid-dev01 puppet-user[29407]: (/Stage[main]/Openstackid/Package[php7.2-mysqlnd]/ensure) ensure changed 'purged' to 'present' is the only thing I notice, not sure if expected | 23:23 |
| *** slaweq has quit IRC | 23:23 | |
| clarkb | corvus: no they are attached to the instance and you get the security group called 'default' by default if the cloud has security groups enabled | 23:23 |
| corvus | hrm. since we put a mirror in this region, i assume we would have updated the 'default' group | 23:23 |
| clarkb | if you server show $instance you should get the security groups listed there then can security-group list/show iirc | 23:24 |
| clarkb | corvus: that is a good point | 23:24 |
| corvus | i'll run it again with --keep --verbose | 23:24 |
| corvus | --verbose was a bad idea | 23:24 |
| clarkb | ha | 23:24 |
| corvus | that's apparently for debugging openstacksdk. | 23:24 |
| corvus | RESP BODY: {"security_groups": [{"rules": [{"from_port": null, "group": {}, "ip_protocol": null, "to_port": null, "parent_group_id": "5a7c1b3d-682e-432b-9d71-fa7dcb8ff89a", "ip_range": {"cidr": "0.0.0.0/0"}, "id": "3839193d-3d6d-4846-9901-13c9788f70f1"}, {"from_port": null, "group": {}, "ip_protocol": null, "to_port": null, "parent_group_id": "5a7c1b3d-682e-432b-9d71-fa7dcb8ff89a", "ip_range": | 23:25 |
| corvus | {"cidr": "::/0"}, "id": "d40bfd91-d467-4af1-a887-ee502d45c5b0"}], "tenant_id": "462ecebbb6e34add9eeeae3936aa6cb9", "id": "5a7c1b3d-682e-432b-9d71-fa7dcb8ff89a", "name": "default", "description": "Default security group"}]} | 23:25 |
| corvus | though it did show me that. | 23:25 |
| clarkb | that rules out security group issues I think. The null port values mean any port and the two rules cover ipv4 and ipv6 | 23:26 |
| clarkb | (and you listed allowed things only no blocking rules) | 23:26 |
| *** markvoelker has joined #openstack-infra | 23:27 | |
| *** lefteri5 has quit IRC | 23:27 | |
| *** mattw4 has quit IRC | 23:28 | |
| clarkb | fungi: that package install request aws added to puppet a while back | 23:28 |
| clarkb | so I'm not sure why it would've been purged before, but we definitely ask to install it | 23:29 |
| corvus | clarkb: do you see anything about sshd in http://paste.openstack.org/show/745777/ ? | 23:29 |
| *** ekultails has quit IRC | 23:31 | |
| openstackgerrit | Merged openstack-infra/puppet-kerberos master: Simplify service management https://review.openstack.org/638777 | 23:31 |
| clarkb | corvus: no i also see little in the way of configuring ens3 or cloud-init | 23:32 |
| clarkb | possible mnaser's image isn't set up right? | 23:32 |
| corvus | mnaser: i'm trying to start a v2-highcpu-8 instance with image "Ubuntu 18.04 LTS (x86_64) [2018-08-23]" in sjc1 and i can't connect to port 22 and the console log doesn't show anything about sshd. instance 6c7d1634-6bda-41bb-a3cf-0f3947b04d29 if that helps. | 23:34 |
| clarkb | fungi: https://review.openstack.org/#/c/616002/9 is probably safe to approve? we can merge that modnay morning when smarcet and you are likely to be around a bit more though other than that unexpected but expected package isntall -dev looks fine | 23:35 |
| clarkb | up to you. Though I've been given a hard stop at 5pm local time to cook dinner | 23:36 |
| clarkb | (I half suspect you've found your weekend already) | 23:36 |
| clarkb | corvus: we could upload our own image (maybe just upstream ubuntu image?) and try that? though chances are that is where mnaser got the image too | 23:37 |
| *** wolverineav has joined #openstack-infra | 23:38 | |
| *** wolverineav has quit IRC | 23:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP Run zuul-preview https://review.openstack.org/637654 | 23:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Make kdc03 the master kerberos kdc and admin server https://review.openstack.org/638793 | 23:51 |
| clarkb | I'm going to WIP ^ and keep it WIP until ready to go through the steps described in the commit message | 23:51 |
| *** mriedem has quit IRC | 23:52 | |
| *** rascasoft has joined #openstack-infra | 23:58 | |
| *** markvoelker has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!