| *** Swami has quit IRC | 00:02 | |
| *** odyssey4me has quit IRC | 00:02 | |
| *** odyssey4me has joined #openstack-infra | 00:03 | |
| bmace | clarkb: i'll be happy to let you know if i end up being a deviation from the norm, lol | 00:03 |
|---|---|---|
| *** rlandy has quit IRC | 00:03 | |
| *** felipemonteiro__ has joined #openstack-infra | 00:07 | |
| corvus | clarkb: we don't have a way of only doing git:// for some hosts; it's all or none (ie, we can't virtualhost git://) | 00:07 |
| *** Guest85132 has joined #openstack-infra | 00:08 | |
| clarkb | corvus: ya we can only restrict it to path. So we can prevent it for working for git.zuul-ci.org/zuul but not for git.zuul-ci.org/openstack/nova if we let openstack keep current setup | 00:08 |
| clarkb | corvus: but I think that will be far less likely to confuse and existing openstack projcet simply because they will be more aware of the history in the hosting environment so I think that is ok? | 00:08 |
| clarkb | I really doubt many users would ever notice | 00:09 |
| clarkb | but if we just look at devstack for example it is all git:// by default :( | 00:10 |
| clarkb | turning that off will require a bit of work | 00:10 |
| corvus | clarkb: i think i understand -- you're saying: don't make symlinks for all projects. only make symlinks for (eg) zuul. point zuul apache vhost at symlinks. the only "problem" is that still means git://git.zuul-ci.org/openstack/nova still works. | 00:10 |
| clarkb | corvus: ya | 00:11 |
| corvus | switching devstack should be easy enough :) we should do it anyway | 00:11 |
| clarkb | ++ but I can just imagine that means people are using git:// all over the place via cargo culting | 00:11 |
| clarkb | (we can watch it with haproxy though and tell people nicely to stop | 00:11 |
| corvus | yep, it shouldn't be a mystery | 00:12 |
| clarkb | but that way a zuul user will never be confused as to why zuul can come from "two" places | 00:13 |
| clarkb | (a dns lookup should probably sort it out but...) | 00:13 |
| *** Guest85132 has quit IRC | 00:13 | |
| openstackgerrit | Merged openstack-infra/system-config master: Properly deprecate stackforge https://review.openstack.org/554312 | 00:14 |
| dmsimard | clarkb: I got stuck in some meetings and other things today. Trying the mirror now. | 00:18 |
| *** danpawlik has joined #openstack-infra | 00:18 | |
| *** Krenair_ has joined #openstack-infra | 00:21 | |
| *** danpawlik has quit IRC | 00:22 | |
| *** dingyichen has joined #openstack-infra | 00:25 | |
| *** salv-orlando has joined #openstack-infra | 00:27 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove tripleo jobs https://review.openstack.org/555037 | 00:31 |
| *** salv-orlando has quit IRC | 00:32 | |
| dmsimard | Seems like the lowest version of shade that the limestone cloud will work on with launch-node is 1.22.2 | 00:34 |
| dmsimard | I tried most of the lower versions (but skipped 1.22.0 and 1.22.1) | 00:34 |
| dmsimard | Lower versions had stack traces such as http://paste.openstack.org/show/708315/ or http://paste.openstack.org/show/708317/ | 00:36 |
| dmsimard | Still have that unhappy openstack inventory list however.. | 00:38 |
| pabelanger | what is the error? I just launched new servers today without issue | 00:38 |
| pabelanger | what version of shade it is using? | 00:38 |
| dmsimard | pabelanger: the launch-env as documented is pinned to Ansible 2.1 and Shade 1.11 | 00:39 |
| dmsimard | However, that version of shade is too old to work with the limestone cloud -- it uses nova for floating IPs and I guess that's harder than neutron or something | 00:39 |
| pabelanger | dmsimard: is /etc/ansible/hosts/openstack failing? | 00:40 |
| dmsimard | pabelanger: it's not, it's just apparently just never returning | 00:40 |
| corvus | dmsimard, clarkb: i don't think we've discussed backing up the zuul build database. i think we should consider it transient data, and i don't think it's worth it. in fact, i think we need to start purging old data. | 00:40 |
| *** felipemonteiro__ has quit IRC | 00:40 | |
| dmsimard | corvus: I'd love to keep the sql reporter data, it's pretty valuable for detailed reporting and metrics | 00:41 |
| corvus | dmsimard: you want to keep all the data, forever? | 00:41 |
| pabelanger | dmsimard: calling it directly using OS shade, works as expected | 00:42 |
| corvus | dmsimard: that sounds like data warehousing, and it's a very different process from anything else we do. | 00:42 |
| pabelanger | shade==1.27.1 | 00:42 |
| pabelanger | is listed with pip freeze | 00:42 |
| *** diablo_rojo has quit IRC | 00:43 | |
| clarkb | corvus: I'm good with that. Similar to our other job related data (logs graphite logstash subunit2sql) | 00:43 |
| corvus | i think we should keep some number of months worth of data. perhaps 6 or 12. i haven't really looked at the volume yet to figure out what's reasonable. | 00:44 |
| *** hongbin has joined #openstack-infra | 00:44 | |
| corvus | but off the top of my head, a year's worth is probably about 7 million rows in the builds table. | 00:44 |
| dmsimard | corvus: I don't think it's realistic to keep the data forever but I wouldn't have a 31 day rotation like we do for logs | 00:44 |
| corvus | dmsimard: oh, yes, certainly more than that :) | 00:44 |
| dmsimard | Last number I had for monthly jobs was ~300k | 00:44 |
| dmsimard | that was around the time we had the zuul v3 migration | 00:45 |
| *** diablo_rojo has joined #openstack-infra | 00:45 | |
| dmsimard | I could look again if only I had mysql-client :D | 00:45 |
| dmsimard | pabelanger: what version of ansible ? | 00:45 |
| corvus | so that's half my estimate. we agree on order of magnitude (which is really probably all that's important right now) | 00:45 |
| *** zhurong has joined #openstack-infra | 00:45 | |
| corvus | dmsimard: i think it's fine to install mysql-client on scheduler. doing that in puppet-zuul sounds fine. | 00:46 |
| pabelanger | dmsimard: we use 2.2.1.0 | 00:46 |
| dmsimard | pabelanger: ok, I'll upgrade from 2.1.1.0 which is pinned in launch-env and try again. | 00:46 |
| dmsimard | corvus: ack, I'll send a patch. | 00:47 |
| mnaser | pabelanger: sorry I never got around answering, probably not a good time to bump things up because I will be fairly afk next week | 00:47 |
| pabelanger | dmsimard: yah, might be worth a test | 00:47 |
| pabelanger | mnaser: np! | 00:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Install the mysql-client package on the scheduler whne using MySQL https://review.openstack.org/555102 | 00:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Install the mysql-client package on the scheduler when using MySQL https://review.openstack.org/555102 | 00:50 |
| *** agopi|dinner has joined #openstack-infra | 00:51 | |
| *** danpawlik has joined #openstack-infra | 00:52 | |
| *** yamamoto has joined #openstack-infra | 00:52 | |
| *** danpawlik has quit IRC | 00:57 | |
| *** yamamoto has quit IRC | 00:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [WIP] Test growroot in boot tests https://review.openstack.org/555103 | 00:59 |
| dmsimard | pabelanger: same thing with ansible 2.2 and latest shade. It's really weird. It gets stuck on "Running: ['/etc/ansible/hosts/openstack', '--list']". I'll add myself some debugging in there. | 01:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs master: Amend top-level project hosting spec https://review.openstack.org/555104 | 01:12 |
| pabelanger | dmsimard: and your using default clouds.yaml file from /etc/openstack? | 01:13 |
| pabelanger | dmsimard: I've seen times when we have wrong credentials or provider is down, for it to not work | 01:14 |
| dmsimard | mordred: why does the openstack inventory care about images ? http://paste.openstack.org/show/708378/ | 01:14 |
| ianw | dmsimard: also, if it's the all-clouds thing then it's querying everything and it takes forever, it might not be stuck | 01:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/jeepyb master: Support cgit alias sites https://review.openstack.org/555105 | 01:15 |
| dmsimard | ianw: I'll double check, thanks | 01:17 |
| pabelanger | odd, I don't see those debug messages when I run it | 01:17 |
| dmsimard | wow uh hrm | 01:17 |
| dmsimard | We have a *LOT* of images in rax dfw | 01:17 |
| dmsimard | but that looks to be the nodepool tenant.. | 01:18 |
| corvus | clarkb, fungi, mordred: https://review.openstack.org/#/q/topic:git-alias contains both approaches for git hosting. | 01:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Unpause Centos 7 for dib 2.12.1 https://review.openstack.org/555106 | 01:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Unpause all builds for dib 2.12.1 https://review.openstack.org/555107 | 01:19 |
| pabelanger | dmsimard: ianw: I wouldn't thing you'd need all-clouds to launch the mirror | 01:20 |
| dmsimard | $ openstack --os-cloud openstackjenkins-rax --os-region DFW image list |wc -l | 01:20 |
| dmsimard | 3238 | 01:20 |
| dmsimard | pabelanger: I'm not (willingly) using all-clouds, at least afaict | 01:20 |
| dmsimard | It's probably using the default clouds.yaml from /etc/openstack but I'll double check | 01:20 |
| pabelanger | yah, I image you just export OS_CLOUD=openstackzuul-limestone | 01:21 |
| pabelanger | before using launch-node.py | 01:21 |
| pabelanger | or openstackci-linestone | 01:21 |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Revert "Use -python35-jobs-horizon for trove-dashboard" https://review.openstack.org/555108 | 01:21 |
| dmsimard | pabelanger: yes | 01:22 |
| dmsimard | pabelanger: the node launches fine in the right cloud, it's really just the ansible inventory list | 01:22 |
| dmsimard | pabelanger: also -- re: you not debug messages might be because of cache | 01:22 |
| dmsimard | I suspect we're not doing a good job at keeping images tidy, there's way too many images.. just in rax-dfw there's 3.2k images and for some reason the openstack inventory is going through all of them | 01:23 |
| pabelanger | dmsimard: yah, but why are you not using cache? | 01:23 |
| dmsimard | pabelanger: because launch-node regenerates the cache :P | 01:23 |
| ianw | oh i'm 100% certain i hit this for some reason | 01:23 |
| ianw | https://review.openstack.org/#/c/529187/ is possibly related | 01:24 |
| clarkb | but th cache generation should never use the all clouds file | 01:24 |
| clarkb | it should only use the control plane clouds.yaml | 01:24 |
| pabelanger | ianw: +3 | 01:25 |
| *** liusheng has quit IRC | 01:26 | |
| pabelanger | EOD now, but I can help on mirror tomorrow morning if you want more eyes dmsimard | 01:26 |
| *** liusheng has joined #openstack-infra | 01:27 | |
| dmsimard | amount of images in rax-dfw (openstackjenkins-rax) | 01:27 |
| dmsimard | http://paste.openstack.org/raw/708402/ | 01:27 |
| clarkb | dmsimard: are any recent? | 01:27 |
| *** salv-orlando has joined #openstack-infra | 01:27 | |
| clarkb | we had leakage problems.before zk I wonder if that persists now with zk | 01:28 |
| ianw | dmsimard: ok http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-25.log.html#t2018-01-25T00:38:53 is what i'm thinking of when i had this issue | 01:28 |
| clarkb | the names have epoch timestamps in them iirc | 01:28 |
| dmsimard | clarkb: https://paste.fedoraproject.org/paste/fS395oQcgGnQLstbVusZ6g | 01:28 |
| dmsimard | clarkb: a more readable version https://paste.fedoraproject.org/paste/YE~N562IeIraANwn3lNKcw/raw | 01:30 |
| ianw | dmsimard: there was an invalid entry in there, and it was quite hard to debug | 01:30 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix default partition type https://review.openstack.org/554771 | 01:30 |
| dmsimard | some images go as far back as ... march 8th | 01:30 |
| *** harlowja has quit IRC | 01:31 | |
| dmsimard | if nodepool v3 still has alien images lookup we could probably use that to clean up the delta | 01:31 |
| *** danpawlik has joined #openstack-infra | 01:31 | |
| *** cshastri has joined #openstack-infra | 01:32 | |
| clarkb | I've just cleaned them up by removing the current two from the list back when we leaked often. Looking at timestamps I think zk probably did mostly correct the issue | 01:32 |
| *** salv-orlando has quit IRC | 01:32 | |
| clarkb | (zk can track the state far better than mysql + gearman) | 01:32 |
| dmsimard | clarkb: not sure I understand "current two from the list" | 01:32 |
| clarkb | dmsimard: nodepool always has at least one but typically two current valid images | 01:33 |
| clarkb | if it is running | 01:33 |
| dmsimard | right | 01:33 |
| clarkb | so remove the current two images of each type then tell openstack to delete the others | 01:33 |
| dmsimard | alien-image-list is supposed to list all the images which aren't "current" | 01:33 |
| dmsimard | so you can safely delete them | 01:34 |
| clarkb | I think our multiple providers per cloud prevented that from being reliable for us in the past | 01:34 |
| clarkb | but we dont do that anymore | 01:34 |
| dmsimard | the command is running right now but it's (probably understandably) not returning very fast, I'll go grab a drink. | 01:34 |
| dmsimard | ianw: seriously though, I have no clue why the openstack inventory would care about images | 01:35 |
| dmsimard | I don't see image things in https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/openstack.py | 01:35 |
| *** danpawlik has quit IRC | 01:36 | |
| pabelanger | I think it is shade, but default will learn about them | 01:36 |
| clarkb | dmsimard: its probably so that you can extract info about the image vms are booted on | 01:36 |
| ianw | clarkb: 2.12.1 pushed ... once it roles out i think do centos first for sanity https://review.openstack.org/555106 then the rest. i left the "pause: " lines in because it's handy to know where to flip that | 01:36 |
| clarkb | and so that you know what you can make new VMs from | 01:36 |
| clarkb | ianw: sounds good to me | 01:36 |
| pabelanger | now I EOD for real | 01:37 |
| clarkb | dmsimard: but you nees to figure out why launch node even looked at those clouds | 01:37 |
| clarkb | it should not | 01:37 |
| dmsimard | clarkb: that's probably fair... but going and doing a get for each and every image ? the equivalent of "server list" | 01:37 |
| dmsimard | er, I didn't finish my sentence | 01:37 |
| dmsimard | the equivalent of "server list" provides the image used | 01:37 |
| clarkb | dmsimard: only the name and uuid is provide from that | 01:38 |
| dmsimard | ugh | 01:38 |
| dmsimard | I had a clouds.yaml in ~/.config/openstack/ | 01:39 |
| clarkb | the git log may provide the justification from whoever added it | 01:39 |
| dmsimard | from testing the limestone stuff, which I probably took from all-clouds I guess | 01:39 |
| dmsimard | ok... let's try that again | 01:39 |
| *** wolverineav has quit IRC | 01:43 | |
| *** wolverineav has joined #openstack-infra | 01:44 | |
| *** liujiong has joined #openstack-infra | 01:46 | |
| *** danpawlik has joined #openstack-infra | 01:47 | |
| *** ykarel|away has joined #openstack-infra | 01:48 | |
| *** wolverineav has quit IRC | 01:48 | |
| *** liujiong has quit IRC | 01:50 | |
| *** danpawlik has quit IRC | 01:51 | |
| dmsimard | ok I give up for tonight, now shade can't find floating IPs but they show up in "openstack floating ip list" | 01:52 |
| *** yamamoto has joined #openstack-infra | 01:53 | |
| *** ramishra has joined #openstack-infra | 01:55 | |
| *** yamamoto has quit IRC | 01:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run expand-groups.sh occasionally under cron https://review.openstack.org/529187 | 02:02 |
| *** zhurong has quit IRC | 02:02 | |
| *** mriedem has quit IRC | 02:07 | |
| *** danpawlik has joined #openstack-infra | 02:09 | |
| *** dtruong2 has joined #openstack-infra | 02:09 | |
| *** wolverineav has joined #openstack-infra | 02:10 | |
| *** danpawlik has quit IRC | 02:13 | |
| *** agopi|dinner is now known as agopi | 02:14 | |
| *** wolverineav has quit IRC | 02:14 | |
| *** jlabarre has quit IRC | 02:15 | |
| *** wolverineav has joined #openstack-infra | 02:16 | |
| *** owalsh_ has joined #openstack-infra | 02:16 | |
| *** owalsh has quit IRC | 02:20 | |
| *** HeOS has quit IRC | 02:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Unpause Centos 7 for dib 2.12.1 https://review.openstack.org/555106 | 02:24 |
| *** salv-orlando has joined #openstack-infra | 02:28 | |
| *** dtruong2 has quit IRC | 02:29 | |
| *** salv-orlando has quit IRC | 02:34 | |
| *** yamamoto has joined #openstack-infra | 02:37 | |
| *** edmondsw has joined #openstack-infra | 02:42 | |
| *** danpawlik has joined #openstack-infra | 02:44 | |
| *** ykarel|away has quit IRC | 02:45 | |
| *** yamamoto has quit IRC | 02:46 | |
| *** wolverineav has quit IRC | 02:46 | |
| *** andreas_s has joined #openstack-infra | 02:48 | |
| *** danpawlik has quit IRC | 02:49 | |
| *** edmondsw has quit IRC | 02:49 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-infra | 02:52 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-infra | 02:52 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-infra | 02:53 | |
| *** germs has quit IRC | 02:53 | |
| *** germs has joined #openstack-infra | 02:53 | |
| *** andreas_s has quit IRC | 02:53 | |
| *** psachin has joined #openstack-infra | 02:56 | |
| *** yamamoto has joined #openstack-infra | 02:56 | |
| *** zhurong has joined #openstack-infra | 02:57 | |
| *** shu-mutow has joined #openstack-infra | 03:05 | |
| *** sree has joined #openstack-infra | 03:09 | |
| *** dave-mccowan has quit IRC | 03:11 | |
| *** cshastri has quit IRC | 03:13 | |
| prometheanfire | pabelanger: what do we expect `ifup INTERFACE` to do in https://github.com/prometheanfire/glean/blob/master/glean/init/glean%40.service#L13 run dhcp? | 03:14 |
| prometheanfire | pabelanger: I ask because gentoo doesn't have an ifup command | 03:14 |
| *** eernst has joined #openstack-infra | 03:14 | |
| *** annp has joined #openstack-infra | 03:14 | |
| *** danpawlik has joined #openstack-infra | 03:18 | |
| persia | prometheanfire: Out of context, `ifup` reads /etc/network/interfaces (debian style network configuration file) and configures the listed interfaces as specified. A common base configuration is to run dhcp on each interface. Gentoo equivalent might be something lile `/etc/init.d/net.%I start` if I read the docs right. | 03:22 |
| prometheanfire | for openrc, yes | 03:22 |
| prometheanfire | persia: I'd like to convert it to networkd if possible | 03:23 |
| *** danpawlik has quit IRC | 03:23 | |
| ianw | prometheanfire: from memory, glean has put in /etc/network/interfaces/eth-watherver.cfg, and that brings it up | 03:24 |
| prometheanfire | hmm | 03:25 |
| prometheanfire | that didn't happen for gentoo at least (for good reason I guess) | 03:25 |
| *** cshastri has joined #openstack-infra | 03:27 | |
| persia | prometheanfire: Tricky. Maybe something like `systemctl start network.service`? I can't find any way to start just one interface with systemd-networkd. | 03:27 |
| prometheanfire | maybe? | 03:27 |
| persia | Or maybe `systemctl reload network.service` if you'd edited the file. | 03:27 |
| prometheanfire | the simple change would be to just dhclient the interface | 03:27 |
| persia | Doesn't glean consume static information if provided before falling back to DHCP? | 03:30 |
| prometheanfire | it does | 03:30 |
| *** hongbin has quit IRC | 03:30 | |
| *** salv-orlando has joined #openstack-infra | 03:30 | |
| *** harlowja has joined #openstack-infra | 03:32 | |
| openstackgerrit | Merged openstack/diskimage-builder master: remove portage git directory https://review.openstack.org/553914 | 03:32 |
| *** dhajare has joined #openstack-infra | 03:33 | |
| *** salv-orlando has quit IRC | 03:35 | |
| *** d0ugal_ has joined #openstack-infra | 03:38 | |
| persia | Then dhclient doesn't do the same thing, I think. | 03:39 |
| *** d0ugal has quit IRC | 03:39 | |
| ianw | new centos images are uploaded, just waiting on something to boot one | 03:50 |
| *** danpawlik has joined #openstack-infra | 03:51 | |
| prometheanfire | who owns glean? | 03:53 |
| *** harlowja has quit IRC | 03:54 | |
| * prometheanfire wants to talk to the 'ptl' :P | 03:54 | |
| *** danpawlik has quit IRC | 03:56 | |
| ianw | that's more like it "/dev/vda1 75G 7.8G 64G 11% /" | 03:56 |
| prometheanfire | fdisk? | 03:58 |
| prometheanfire | I did verify it was good locally | 03:58 |
| prometheanfire | 21:48 < prometheanfire > /dev/vda1 * 2048 41943006 41940959 20G 83 Linux | 03:58 |
| *** udesale has joined #openstack-infra | 04:03 | |
| *** edmondsw has joined #openstack-infra | 04:05 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [WIP] Test growroot in boot tests https://review.openstack.org/555103 | 04:07 |
| *** dhajare has quit IRC | 04:09 | |
| *** edmondsw has quit IRC | 04:10 | |
| *** persia has quit IRC | 04:13 | |
| *** persia has joined #openstack-infra | 04:14 | |
| *** ykarel|away has joined #openstack-infra | 04:22 | |
| *** iyamahat has quit IRC | 04:27 | |
| *** s-shiono has joined #openstack-infra | 04:31 | |
| *** salv-orlando has joined #openstack-infra | 04:31 | |
| *** salv-orlando has quit IRC | 04:36 | |
| *** zhurong has quit IRC | 04:45 | |
| *** felipemonteiro__ has joined #openstack-infra | 04:45 | |
| *** pgadiya has joined #openstack-infra | 04:46 | |
| *** germs has quit IRC | 04:49 | |
| *** germs has joined #openstack-infra | 04:49 | |
| *** germs has quit IRC | 04:49 | |
| *** germs has joined #openstack-infra | 04:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Unpause all builds for dib 2.12.1 https://review.openstack.org/555107 | 04:52 |
| *** dhajare has joined #openstack-infra | 04:53 | |
| *** germs has quit IRC | 04:54 | |
| *** felipemonteiro__ has quit IRC | 04:57 | |
| *** isssp has joined #openstack-infra | 05:03 | |
| *** ethfci has quit IRC | 05:04 | |
| *** burned has quit IRC | 05:06 | |
| *** lpetrut has joined #openstack-infra | 05:08 | |
| *** isssp has quit IRC | 05:09 | |
| *** agopi_ has joined #openstack-infra | 05:12 | |
| *** isssp has joined #openstack-infra | 05:12 | |
| *** ykarel|away is now known as ykarel | 05:13 | |
| *** agopi has quit IRC | 05:13 | |
| *** imacdonn has quit IRC | 05:15 | |
| *** imacdonn has joined #openstack-infra | 05:15 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: install sudo in the devuser element https://review.openstack.org/555149 | 05:18 |
| *** rosmaita has quit IRC | 05:18 | |
| *** agopi_ is now known as agopi_out | 05:19 | |
| *** gyankum has joined #openstack-infra | 05:26 | |
| *** jaosorior_ is now known as jaosorior | 05:26 | |
| *** salv-orlando has joined #openstack-infra | 05:29 | |
| *** wolverineav has joined #openstack-infra | 05:30 | |
| *** rcernin has quit IRC | 05:34 | |
| *** zhurong has joined #openstack-infra | 05:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add a job build button https://review.openstack.org/554839 | 05:36 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: job-trigger: add initial driver and event https://review.openstack.org/555153 | 05:36 |
| *** udesale_ has joined #openstack-infra | 05:36 | |
| *** udesale__ has joined #openstack-infra | 05:37 | |
| *** eernst has quit IRC | 05:39 | |
| *** udesale has quit IRC | 05:39 | |
| *** udesale_ has quit IRC | 05:41 | |
| *** wolverineav has quit IRC | 05:41 | |
| *** wolverineav has joined #openstack-infra | 05:42 | |
| *** wolverineav has quit IRC | 05:46 | |
| *** rcernin has joined #openstack-infra | 05:46 | |
| *** links has joined #openstack-infra | 05:53 | |
| *** edmondsw has joined #openstack-infra | 05:54 | |
| *** edmondsw has quit IRC | 05:59 | |
| *** lpetrut has quit IRC | 06:03 | |
| *** pcichy has joined #openstack-infra | 06:03 | |
| *** rcernin has quit IRC | 06:06 | |
| *** rcernin has joined #openstack-infra | 06:08 | |
| *** lpetrut has joined #openstack-infra | 06:12 | |
| *** iyamahat has joined #openstack-infra | 06:15 | |
| *** e0ne has joined #openstack-infra | 06:15 | |
| *** lucas-afk has quit IRC | 06:20 | |
| *** lucasagomes has joined #openstack-infra | 06:23 | |
| *** threestrands has quit IRC | 06:25 | |
| *** lathiat has quit IRC | 06:30 | |
| *** dbecker has quit IRC | 06:32 | |
| *** e0ne has quit IRC | 06:33 | |
| *** agopi_out has quit IRC | 06:33 | |
| *** lpetrut has quit IRC | 06:33 | |
| *** zhurong has quit IRC | 06:34 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Temporarily disable py35 jobs for trove-dashboard stable branches https://review.openstack.org/555108 | 06:45 |
| *** armaan has joined #openstack-infra | 06:46 | |
| *** e0ne has joined #openstack-infra | 06:46 | |
| *** iyamahat has quit IRC | 06:46 | |
| *** iyamahat has joined #openstack-infra | 06:46 | |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Temporarily disable py35 jobs for trove-dashboard stable branches https://review.openstack.org/555108 | 06:47 |
| *** udesale_ has joined #openstack-infra | 06:48 | |
| *** udesale__ has quit IRC | 06:50 | |
| *** e0ne has quit IRC | 06:52 | |
| *** claudiub has joined #openstack-infra | 06:57 | |
| *** pcichy has quit IRC | 06:58 | |
| *** masber has quit IRC | 07:05 | |
| *** masber has joined #openstack-infra | 07:06 | |
| *** pcichy has joined #openstack-infra | 07:11 | |
| *** d0ugal_ has quit IRC | 07:14 | |
| *** zhurong has joined #openstack-infra | 07:14 | |
| *** d0ugal has joined #openstack-infra | 07:14 | |
| *** d0ugal has quit IRC | 07:14 | |
| *** d0ugal has joined #openstack-infra | 07:14 | |
| *** links has quit IRC | 07:15 | |
| *** jtomasek has joined #openstack-infra | 07:17 | |
| *** jtomasek has quit IRC | 07:17 | |
| *** jtomasek has joined #openstack-infra | 07:18 | |
| *** iyamahat has quit IRC | 07:19 | |
| *** alexchadin has joined #openstack-infra | 07:19 | |
| *** andreas_s has joined #openstack-infra | 07:21 | |
| *** links has joined #openstack-infra | 07:23 | |
| *** andreas_s has quit IRC | 07:25 | |
| *** claudiub has quit IRC | 07:25 | |
| *** isssp has quit IRC | 07:25 | |
| *** diablo_rojo has quit IRC | 07:25 | |
| *** jaosorior has quit IRC | 07:25 | |
| *** trown|outtypewww has quit IRC | 07:25 | |
| *** vaidy has quit IRC | 07:25 | |
| *** pgaxatte has quit IRC | 07:25 | |
| *** slaweq has quit IRC | 07:25 | |
| *** sambetts|afk has quit IRC | 07:25 | |
| *** HenryG has quit IRC | 07:25 | |
| *** hrybacki has quit IRC | 07:25 | |
| *** petems has quit IRC | 07:25 | |
| *** edwarnicke has quit IRC | 07:25 | |
| *** seongsoocho has quit IRC | 07:25 | |
| *** rajinir has quit IRC | 07:25 | |
| *** Anticimex has quit IRC | 07:25 | |
| *** jungleboyj has quit IRC | 07:25 | |
| *** puiterwijk has quit IRC | 07:25 | |
| *** rajinir_ has joined #openstack-infra | 07:26 | |
| *** rajinir_ has quit IRC | 07:26 | |
| *** rajinir_ has joined #openstack-infra | 07:26 | |
| *** rajinir_ is now known as rajinir | 07:26 | |
| *** HenryG_ has joined #openstack-infra | 07:26 | |
| *** slaweq has joined #openstack-infra | 07:26 | |
| *** hrybacki has joined #openstack-infra | 07:26 | |
| *** petems has joined #openstack-infra | 07:26 | |
| *** jaosorior has joined #openstack-infra | 07:26 | |
| *** pgaxatte has joined #openstack-infra | 07:26 | |
| *** isssp has joined #openstack-infra | 07:26 | |
| *** puiterwijk has joined #openstack-infra | 07:27 | |
| *** seongsoocho has joined #openstack-infra | 07:27 | |
| *** jungleboyj has joined #openstack-infra | 07:27 | |
| *** andreas_s has joined #openstack-infra | 07:27 | |
| *** edwarnicke_ has joined #openstack-infra | 07:27 | |
| *** diablo_rojo has joined #openstack-infra | 07:28 | |
| *** wznoinsk has quit IRC | 07:28 | |
| *** vaidy has joined #openstack-infra | 07:28 | |
| *** trown has joined #openstack-infra | 07:28 | |
| *** gagehugo has quit IRC | 07:28 | |
| *** ruhe has quit IRC | 07:29 | |
| *** sambetts_ has joined #openstack-infra | 07:29 | |
| *** ruhe has joined #openstack-infra | 07:31 | |
| *** rcernin has quit IRC | 07:31 | |
| *** wznoinsk has joined #openstack-infra | 07:31 | |
| *** eroux has joined #openstack-infra | 07:33 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** gagehugo has joined #openstack-infra | 07:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: job-trigger: add initial driver and event https://review.openstack.org/555153 | 07:39 |
| *** ralonsoh has joined #openstack-infra | 07:39 | |
| *** edmondsw has joined #openstack-infra | 07:42 | |
| *** pcaruana has joined #openstack-infra | 07:42 | |
| *** HeOS has joined #openstack-infra | 07:43 | |
| *** pcaruana has quit IRC | 07:44 | |
| *** pcaruana has joined #openstack-infra | 07:44 | |
| *** pcaruana has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-infra | 07:45 | |
| *** danpawlik has joined #openstack-infra | 07:45 | |
| *** edmondsw has quit IRC | 07:46 | |
| *** pcaruana has quit IRC | 07:47 | |
| *** armaan has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-infra | 07:47 | |
| *** armaan has joined #openstack-infra | 07:47 | |
| *** pcaruana has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-infra | 07:48 | |
| *** pcaruana has quit IRC | 07:50 | |
| *** pcaruana has joined #openstack-infra | 07:50 | |
| *** masber has quit IRC | 07:51 | |
| *** pcaruana has quit IRC | 07:51 | |
| *** pcaruana has joined #openstack-infra | 07:51 | |
| *** armaan has quit IRC | 07:52 | |
| *** pcaruana has quit IRC | 07:53 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Clarify position of depends-on footer https://review.openstack.org/555185 | 07:53 |
| *** pcaruana has joined #openstack-infra | 07:53 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Clarify location of depends-on footer https://review.openstack.org/555185 | 07:53 |
| *** pcaruana has quit IRC | 07:54 | |
| *** pcaruana has joined #openstack-infra | 07:55 | |
| *** pcaruana has quit IRC | 07:56 | |
| *** aspiers[m] has quit IRC | 07:58 | |
| *** bgmccollum has quit IRC | 07:58 | |
| *** ispp has joined #openstack-infra | 07:58 | |
| *** asettle has quit IRC | 07:59 | |
| *** bgmccollum has joined #openstack-infra | 07:59 | |
| *** andymccr has quit IRC | 07:59 | |
| *** Roamer` has quit IRC | 07:59 | |
| *** petrovich has quit IRC | 07:59 | |
| *** isssp has quit IRC | 08:00 | |
| *** Roamer` has joined #openstack-infra | 08:01 | |
| *** lpetrut has joined #openstack-infra | 08:03 | |
| *** lpetrut_ has joined #openstack-infra | 08:04 | |
| *** lpetrut has quit IRC | 08:04 | |
| *** andymccr has joined #openstack-infra | 08:05 | |
| *** pblaho has joined #openstack-infra | 08:05 | |
| *** asettle has joined #openstack-infra | 08:06 | |
| *** pcaruana has joined #openstack-infra | 08:06 | |
| *** asettle is now known as Guest66969 | 08:06 | |
| *** aspiers[m] has joined #openstack-infra | 08:07 | |
| *** pcaruana has quit IRC | 08:07 | |
| *** pcaruana has joined #openstack-infra | 08:08 | |
| *** pcaruana has quit IRC | 08:09 | |
| *** armaan has joined #openstack-infra | 08:09 | |
| *** pcaruana has joined #openstack-infra | 08:10 | |
| *** pcaruana has quit IRC | 08:10 | |
| *** pcaruana has joined #openstack-infra | 08:11 | |
| *** tesseract has joined #openstack-infra | 08:11 | |
| *** masber has joined #openstack-infra | 08:11 | |
| *** pcaruana has quit IRC | 08:12 | |
| *** pcaruana has joined #openstack-infra | 08:13 | |
| *** dingyichen has quit IRC | 08:14 | |
| *** pcaruana has quit IRC | 08:15 | |
| *** pcaruana has joined #openstack-infra | 08:15 | |
| d0ugal | Is there anything in infra that projects can use for preiodic IRC announcements? | 08:15 |
| d0ugal | or, failing that, is there anywhere a simple IRC bot can be hosted? | 08:16 |
| *** pcaruana has quit IRC | 08:16 | |
| d0ugal | (Context being, Mistral has switched using office hours, rather than weekly meetings. I want something to announce the start of each office hour slot in #openstack-mistral) | 08:16 |
| *** pcaruana has joined #openstack-infra | 08:18 | |
| *** masber has quit IRC | 08:18 | |
| *** tosky has joined #openstack-infra | 08:19 | |
| *** pcaruana has quit IRC | 08:20 | |
| *** pcaruana has joined #openstack-infra | 08:20 | |
| *** pcaruana has quit IRC | 08:21 | |
| *** pcaruana has joined #openstack-infra | 08:21 | |
| *** pcaruana has quit IRC | 08:22 | |
| *** jcoufal has joined #openstack-infra | 08:23 | |
| *** pblaho has quit IRC | 08:24 | |
| *** snapiri has joined #openstack-infra | 08:25 | |
| *** petrovich has joined #openstack-infra | 08:26 | |
| *** alexchadin has quit IRC | 08:27 | |
| *** pcaruana has joined #openstack-infra | 08:29 | |
| *** pcaruana has quit IRC | 08:30 | |
| *** s-shiono has quit IRC | 08:31 | |
| *** salv-orlando has joined #openstack-infra | 08:34 | |
| *** markvoelker has joined #openstack-infra | 08:34 | |
| *** electrofelix has joined #openstack-infra | 08:35 | |
| *** yolanda_ is now known as yolanda | 08:35 | |
| *** pcaruana has joined #openstack-infra | 08:36 | |
| *** pcaruana has quit IRC | 08:37 | |
| *** pcaruana has joined #openstack-infra | 08:38 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add diablo_rojo's key for Storyboard Access https://review.openstack.org/552667 | 08:39 |
| *** pcaruana has quit IRC | 08:39 | |
| *** pcaruana has joined #openstack-infra | 08:40 | |
| *** salv-orlando has quit IRC | 08:40 | |
| *** pcaruana has quit IRC | 08:40 | |
| *** pcaruana has joined #openstack-infra | 08:41 | |
| *** pcaruana has quit IRC | 08:41 | |
| *** florianf has joined #openstack-infra | 08:44 | |
| *** amoralej|off is now known as amoralej | 08:44 | |
| *** jpena|off is now known as jpena | 08:48 | |
| *** jpich has joined #openstack-infra | 08:54 | |
| *** masber has joined #openstack-infra | 08:55 | |
| *** e0ne has joined #openstack-infra | 08:56 | |
| *** pcaruana has joined #openstack-infra | 09:02 | |
| frickler | d0ugal: I don't think there is such a thing yet, if you write or find something, it might get hosted together with our other bots on eavesdrop.o.o | 09:03 |
| *** pcaruana has quit IRC | 09:03 | |
| krenczewski | Is there a chance to setup 2-node zuul job that uses ubuntu on one node ans centos-7 on second node? | 09:03 |
| *** pcaruana has joined #openstack-infra | 09:03 | |
| frickler | d0ugal: note there is also a pending infra specs proposal to rework our IRC setup, that may be relevant, too | 09:03 |
| d0ugal | frickler: thanks, I'll take a look. | 09:04 |
| *** pcaruana has quit IRC | 09:05 | |
| frickler | krenczewski: you should be able to setup a corresponding nodeset in your zuul config, yes. | 09:05 |
| krenczewski | I've tried to specify this in my job description in .zuul.yaml by specifying nodeset similar to what is defined in openstack-infra/openstack-zuul-jobs/tree/zuul.d/nodesets.yaml but zuul complains that `extra keys not allowed @ data['name']` | 09:05 |
| frickler | krenczewski: where did you try that? | 09:05 |
| krenczewski | https://review.openstack.org/#/c/554238 | 09:06 |
| *** e0ne has quit IRC | 09:07 | |
| *** dizquierdo has joined #openstack-infra | 09:08 | |
| frickler | krenczewski: commented on the review | 09:09 |
| krenczewski | frickler: thanks for the comment :-) | 09:10 |
| *** e0ne has joined #openstack-infra | 09:11 | |
| *** ethfci has joined #openstack-infra | 09:11 | |
| *** owalsh_ is now known as owalsh | 09:13 | |
| kashyap | frickler: So...following up from yesterday, using `git review -R` reduces the time to push a patch for me | 09:13 |
| *** gfidente has joined #openstack-infra | 09:14 | |
| *** gfidente has joined #openstack-infra | 09:14 | |
| *** e0ne has quit IRC | 09:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: job-trigger: add initial driver and event https://review.openstack.org/555153 | 09:16 |
| *** zhurong has quit IRC | 09:16 | |
| *** masber has quit IRC | 09:17 | |
| *** jesusaur has quit IRC | 09:20 | |
| slaweq | frickler: hi, do You maybe know how we can remove old "feature/something" branches from neutron repo? | 09:20 |
| slaweq | I guess it should be patch to releases repo but I have no idea what patch exactly :) | 09:21 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [WIP] Test growroot in boot tests https://review.openstack.org/555103 | 09:24 |
| *** gfidente has quit IRC | 09:28 | |
| *** masber has joined #openstack-infra | 09:28 | |
| *** gfidente has joined #openstack-infra | 09:29 | |
| *** jesusaur has joined #openstack-infra | 09:29 | |
| *** kjackal has quit IRC | 09:29 | |
| *** edmondsw has joined #openstack-infra | 09:30 | |
| *** hemna_ has quit IRC | 09:31 | |
| *** shu-mutow is now known as shu-mutow-AWAY | 09:32 | |
| *** alexchadin has joined #openstack-infra | 09:34 | |
| *** edmondsw has quit IRC | 09:35 | |
| *** salv-orlando has joined #openstack-infra | 09:35 | |
| witek | hi, could someone please check the `default_url` setting of StoryBoard service? | 09:37 |
| witek | https://github.com/openstack-infra/storyboard/blob/master/etc/storyboard.conf.sample#L172 | 09:37 |
| witek | when the task is updated by gerrit I see: | 09:37 |
| witek | URL: None#!/story/xxx | 09:38 |
| *** salv-orlando has quit IRC | 09:38 | |
| *** salv-orlando has joined #openstack-infra | 09:38 | |
| persia | witek: Could you link to an example that provides that URL, or is it something in the gerrit plugin logs? | 09:39 |
| *** chrisyang_0660 has joined #openstack-infra | 09:42 | |
| *** pblaho has joined #openstack-infra | 09:44 | |
| witek | persia: http://paste.openstack.org/show/708905/ | 09:44 |
| witek | the code handling this is here: | 09:45 |
| witek | https://github.com/openstack-infra/storyboard/blob/master/storyboard/plugin/email/workers.py#L285 | 09:45 |
| persia | Ah, the *email* notification. Thanks. | 09:45 |
| * persia can't fix it, but does understand the problem now. | 09:45 | |
| frickler | persia: witek: there's a story for that already: https://storyboard.openstack.org/#!/story/2001410 | 09:46 |
| *** Guest66969 is now known as asettle | 09:46 | |
| witek | isn't it just configuration of the service? | 09:46 |
| openstackgerrit | Bernd Bausch proposed openstack-infra/zuul master: Add port to webhook URL in Zuul github driver docu https://review.openstack.org/555225 | 09:47 |
| frickler | slaweq: I'm not sure, maybe some other infra-root can help later | 09:47 |
| frickler | witek: no idea, I just noticed it some time ago, too | 09:48 |
| slaweq | frickler: sure, I will ask | 09:48 |
| frickler | kashyap: would be interesting to know how much reduction you see, like 50%, more, less? | 09:48 |
| kashyap | It's like 10 seconds or so | 09:49 |
| kashyap | So 400% improvement or something like that | 09:49 |
| frickler | kashyap: o.k., so back to what you had earlier, iirc | 09:49 |
| kashyap | Yep | 09:49 |
| kashyap | It's just that `git review` WILL fetch & rebase no matter even if you've just rebased | 09:49 |
| kashyap | Probably it's stuck at fetching from 'origin' | 09:50 |
| * kashyap gets back to kicking the stuffing out of unit tests | 09:50 | |
| frickler | kashyap: I think it should fetch from the 'gerrit' remote. but maybe this will help others to find an issue on our side | 09:51 |
| kashyap | Yeah, I will investigate it later in the day after I spend the morning fixing these unit tests | 09:51 |
| openstackgerrit | Bernd Bausch proposed openstack-infra/zuul master: Add port to webhook URL in Zuul github driver docu https://review.openstack.org/555225 | 09:52 |
| witek | persia: frickler: added comment to https://storyboard.openstack.org/#!/story/2001410 | 09:53 |
| *** dayou has joined #openstack-infra | 09:54 | |
| *** bmace has quit IRC | 09:54 | |
| *** bmace has joined #openstack-infra | 09:54 | |
| frickler | witek: indeed the mail that notified me about your comment has the correct reference, good point | 09:54 |
| persia | I suspect it needs a change to http://git.openstack.org/cgit/openstack-infra/puppet-storyboard/tree/templates/storyboard.conf.erb , but I'm not sure precisely what needs changing there to make it work. | 09:55 |
| witek | thanks for pointing, I'll try to add it | 09:56 |
| *** hashar has joined #openstack-infra | 09:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add a job build button https://review.openstack.org/554839 | 10:00 |
| *** dkehn_ has quit IRC | 10:06 | |
| *** kjackal has joined #openstack-infra | 10:06 | |
| *** e0ne has joined #openstack-infra | 10:10 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/puppet-storyboard master: Set default_url for email notification https://review.openstack.org/555237 | 10:11 |
| *** shardy has joined #openstack-infra | 10:15 | |
| *** arxcruz is now known as arxcruz|rover | 10:15 | |
| *** udesale_ has quit IRC | 10:16 | |
| *** HeOS has quit IRC | 10:17 | |
| persia | Ah, I understand. When someone makes a change via the Web UI, the address they had gets used in the email, but that doesn't work for API changes. Excellent catch. | 10:22 |
| *** boden has joined #openstack-infra | 10:23 | |
| *** florianf has quit IRC | 10:25 | |
| *** alexchadin has quit IRC | 10:26 | |
| *** masuberu has joined #openstack-infra | 10:27 | |
| *** sree has quit IRC | 10:30 | |
| *** panda|off is now known as panda | 10:30 | |
| *** sree has joined #openstack-infra | 10:30 | |
| *** masber has quit IRC | 10:30 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** derekh has joined #openstack-infra | 10:33 | |
| *** sree has quit IRC | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:36 | |
| *** zoli is now known as zoli|lunch | 10:39 | |
| *** florianf has joined #openstack-infra | 10:40 | |
| *** pbourke has joined #openstack-infra | 10:43 | |
| *** pcichy has quit IRC | 10:48 | |
| *** sshnaidm has quit IRC | 11:00 | |
| *** dsariel has joined #openstack-infra | 11:01 | |
| *** jesusaur has quit IRC | 11:02 | |
| *** logan- has quit IRC | 11:03 | |
| *** logan- has joined #openstack-infra | 11:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily disable py35 jobs for trove-dashboard stable branches https://review.openstack.org/555108 | 11:10 |
| openstackgerrit | Merged openstack-infra/zuul master: Add port to webhook URL in Zuul github driver docu https://review.openstack.org/555225 | 11:12 |
| *** jesusaur has joined #openstack-infra | 11:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [WIP] Test growroot in boot tests https://review.openstack.org/555103 | 11:21 |
| *** HeOS has joined #openstack-infra | 11:23 | |
| mordred | kashyap: fwiw, you can set a config value globally to make -R the default (I have it set- had forgotten that git review rebases things by default...) | 11:39 |
| mordred | kashyap: gitreview.rebase = False is the setting in quesion | 11:39 |
| kashyap | mordred: Ah, merci! | 11:40 |
| *** zhurong has joined #openstack-infra | 11:40 | |
| *** dizquierdo has quit IRC | 11:40 | |
| mordred | kashyap: also - I have no idea if it will help you, but there is also an option I use, gitreview.usepushurl - which causes it to replace the gerrit remote with setting an alternate pushurl on the origin remote ... should save you from needing to fetch/update both origin and gerrit remotes | 11:41 |
| kashyap | mordred: Like that? | 11:41 |
| kashyap | $> tail -2 ~/.gitconfig | 11:41 |
| kashyap | [gitreview] | 11:41 |
| kashyap | rebase = False | 11:41 |
| mordred | kashyap: yup! | 11:42 |
| kashyap | mordred: Okido; (I don't see the 'usepushurl' in the man page of: $> tail -2 ~/.gitconfig | 11:43 |
| kashyap | [gitreview] | 11:43 |
| kashyap | Err, wrong copy / paste. | 11:43 |
| kashyap | ...in the man page of: git-review-1.25 | 11:43 |
| mordred | kashyap: you are quite correct - it sure is missing | 11:44 |
| mordred | (from the man page - patch coming) | 11:44 |
| kashyap | Thanks! /me goes to make lunch | 11:46 |
| *** udesale has joined #openstack-infra | 11:46 | |
| *** manjeets has quit IRC | 11:49 | |
| *** dklyle has quit IRC | 11:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Document usepushurl in the man page https://review.openstack.org/555271 | 11:51 |
| *** armaan has quit IRC | 11:51 | |
| mordred | kashyap, fungi: ^^ | 11:51 |
| *** armaan has joined #openstack-infra | 11:52 | |
| *** Goneri has joined #openstack-infra | 11:52 | |
| *** masuberu has quit IRC | 11:52 | |
| *** masuberu has joined #openstack-infra | 11:52 | |
| *** sambetts_ is now known as sambetts | 11:54 | |
| *** dizquierdo has joined #openstack-infra | 11:57 | |
| *** zoli|lunch is now known as zoli | 11:58 | |
| kashyap | mordred: Noted a couple of OCD nit-picks; but looks good, so still "+1ed" it, FWIW | 11:59 |
| * kashyap now gets back to the kitchen | 11:59 | |
| *** rfolco|off is now known as rfolco|ruck | 12:02 | |
| *** pblaho has quit IRC | 12:02 | |
| *** pblaho has joined #openstack-infra | 12:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Document usepushurl in the man page https://review.openstack.org/555271 | 12:04 |
| *** amoralej is now known as amoralej|lunch | 12:05 | |
| mordred | kashyap: thanks! those are good OCD nit-picks - fixed | 12:06 |
| *** odyssey4me has quit IRC | 12:07 | |
| *** odyssey4me has joined #openstack-infra | 12:08 | |
| *** rosmaita has joined #openstack-infra | 12:08 | |
| kashyap | Heh | 12:12 |
| *** alexchadin has quit IRC | 12:12 | |
| *** cshastri has quit IRC | 12:13 | |
| *** jlabarre has joined #openstack-infra | 12:14 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| *** alexchadin has joined #openstack-infra | 12:15 | |
| *** lathiat has joined #openstack-infra | 12:15 | |
| *** sree has joined #openstack-infra | 12:17 | |
| *** Goneri has quit IRC | 12:17 | |
| *** alexchadin has quit IRC | 12:18 | |
| *** alexchadin has joined #openstack-infra | 12:18 | |
| *** alexchadin has quit IRC | 12:18 | |
| *** alexchadin has joined #openstack-infra | 12:19 | |
| *** alexchadin has quit IRC | 12:19 | |
| *** alexchadin has joined #openstack-infra | 12:20 | |
| *** alexchadin has quit IRC | 12:20 | |
| *** alexchadin has joined #openstack-infra | 12:21 | |
| *** alexchadin has quit IRC | 12:21 | |
| *** alexchadin has joined #openstack-infra | 12:21 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** alexchadin has joined #openstack-infra | 12:22 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** panda is now known as panda|lunch | 12:23 | |
| *** alexchadin has joined #openstack-infra | 12:23 | |
| *** lucasagomes is now known as lucas-hungry | 12:23 | |
| *** chrisyang_0660 has quit IRC | 12:24 | |
| *** sshnaidm has joined #openstack-infra | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** e0ne has quit IRC | 12:31 | |
| *** alexchad_ has joined #openstack-infra | 12:33 | |
| *** alexchadin has quit IRC | 12:33 | |
| *** markvoelker has quit IRC | 12:34 | |
| *** markvoelker has joined #openstack-infra | 12:34 | |
| *** jcoufal_ has joined #openstack-infra | 12:35 | |
| *** dprince has joined #openstack-infra | 12:36 | |
| *** jcoufal has quit IRC | 12:36 | |
| *** e0ne has joined #openstack-infra | 12:45 | |
| *** sree has quit IRC | 12:47 | |
| *** kgiusti has joined #openstack-infra | 12:48 | |
| *** mriedem has joined #openstack-infra | 12:50 | |
| *** jcoufal_ has quit IRC | 12:51 | |
| frickler | infra-root: tempest-full-py3 seems to be failing hard in stackviz collection since about 6h ago, I did some digging but didn't find anything useful yet, help is appreciated. see http://zuul.openstack.org/builds.html?job_name=tempest-full-py3 | 12:51 |
| dmsimard | frickler: could this have anything to do with the BHS1 issues ? | 12:51 |
| dmsimard | there was an issue with synchronize tasks | 12:52 |
| *** eroux has quit IRC | 12:52 | |
| dmsimard | nope, I see rax in there | 12:52 |
| frickler | dmsimard: no, this new one seems strictly related to python3 somehow | 12:52 |
| dmsimard | I'm about to jump in a meeting so I can't look :( | 12:52 |
| dmsimard | kashyap: haven't fully read backlog but it looks like you've figured your performance issues out ? | 12:53 |
| kashyap | dmsimard: For now, yes. I forgot to give the rebase (-R) to `git review` | 12:53 |
| *** yamamoto has quit IRC | 12:54 | |
| *** electrofelix has quit IRC | 12:58 | |
| *** zoli is now known as zoli|afk | 12:59 | |
| *** zoli|afk is now known as zoli | 12:59 | |
| *** zhurong has quit IRC | 13:00 | |
| *** udesale has quit IRC | 13:00 | |
| *** pgadiya has quit IRC | 13:02 | |
| *** bobh has joined #openstack-infra | 13:06 | |
| openstackgerrit | Doug Hellmann proposed openstack-dev/hacking master: add lower-constraints job https://review.openstack.org/555296 | 13:06 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: add lower-constraints job https://review.openstack.org/555297 | 13:07 |
| dhellmann | I just submitted 9 changes to add lower-constraints jobs. Let me know if that's too many at one time for zuul's configuration resolver. | 13:09 |
| dhellmann | they all depend on https://review.openstack.org/#/c/555034/ which could use another review when someone has time | 13:09 |
| *** Goneri has joined #openstack-infra | 13:10 | |
| *** shardy_ has joined #openstack-infra | 13:14 | |
| *** shardy has quit IRC | 13:16 | |
| *** dhajare has quit IRC | 13:18 | |
| *** udesale has joined #openstack-infra | 13:18 | |
| AJaeger | dhellmann: check http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 13:18 |
| *** dhajare has joined #openstack-infra | 13:18 | |
| AJaeger | dhellmann: if we start swapping, we're dead... | 13:18 |
| rosmaita | mordred: need you to take a look at http://logs.openstack.org/38/554638/3/check/build-openstack-sphinx-docs/f55f79a/ara/file/98fc8620-69f7-4cc6-81ad-a4352394036f/#line-51 , the "run sphinx using legacy pbr runner" role may be broken | 13:18 |
| dhellmann | AJaeger : ack, thanks for that link | 13:19 |
| *** sshnaidm has quit IRC | 13:19 | |
| *** dhajare has quit IRC | 13:20 | |
| *** sshnaidm has joined #openstack-infra | 13:20 | |
| *** dmsimard has quit IRC | 13:21 | |
| *** dtantsur is now known as dtantsur|brb | 13:21 | |
| *** yamamoto has joined #openstack-infra | 13:21 | |
| *** lucas-hungry is now known as lucasagomes | 13:24 | |
| *** panda|lunch is now known as panda | 13:25 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/elastic-recheck master: add query for tempest-full-py3 post failure bug 1758054 https://review.openstack.org/555302 | 13:26 |
| openstack | bug 1758054 in OpenStack-Gate "stackviz failing on tempest-full-py3" [Critical,New] https://launchpad.net/bugs/1758054 | 13:26 |
| *** dizquierdo has quit IRC | 13:26 | |
| prometheanfire | who 'owns' glean? I want to talk about proposed changes to it (make it use networkd) | 13:26 |
| AJaeger | prometheanfire: did you check the core reviewer list? | 13:27 |
| AJaeger | Ah, I see, it'S part of infra... | 13:28 |
| AJaeger | mordred: weren't you involved with glean? ^ | 13:28 |
| *** cshastri has joined #openstack-infra | 13:29 | |
| prometheanfire | AJaeger: mordred sounds right, but it's been a while | 13:30 |
| *** psachin has quit IRC | 13:32 | |
| *** eharney has joined #openstack-infra | 13:32 | |
| prometheanfire | AJaeger: atm it looks like I'd need to write another 'backend' for it to spit out the networkd config files | 13:33 |
| *** eernst has joined #openstack-infra | 13:33 | |
| prometheanfire | there's also the problem that it relies on distro info to select the backend, when that's not the best anymore (ubuntu supports networkd now iirc, so does fedora (and gentoo)) | 13:33 |
| *** germs has joined #openstack-infra | 13:33 | |
| *** germs has quit IRC | 13:33 | |
| *** germs has joined #openstack-infra | 13:33 | |
| *** germs has quit IRC | 13:33 | |
| *** germs has joined #openstack-infra | 13:34 | |
| *** germs has quit IRC | 13:34 | |
| *** germs has joined #openstack-infra | 13:34 | |
| *** dmsimard has joined #openstack-infra | 13:34 | |
| *** dmsimard has quit IRC | 13:34 | |
| *** dmsimard has joined #openstack-infra | 13:34 | |
| *** dhill_ has quit IRC | 13:35 | |
| *** armaan has quit IRC | 13:35 | |
| *** armaan has joined #openstack-infra | 13:36 | |
| *** jcoufal has joined #openstack-infra | 13:36 | |
| *** amoralej|lunch is now known as amoralej | 13:37 | |
| *** pickle has joined #openstack-infra | 13:37 | |
| *** ldnunes has joined #openstack-infra | 13:40 | |
| *** armaan has quit IRC | 13:40 | |
| *** esberglu has joined #openstack-infra | 13:48 | |
| *** trown is now known as trown|brb | 13:49 | |
| *** gouthamr has joined #openstack-infra | 13:50 | |
| *** jcoufal_ has joined #openstack-infra | 13:52 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** trown|brb is now known as trown | 13:54 | |
| *** jcoufal has quit IRC | 13:54 | |
| *** ykarel has quit IRC | 13:54 | |
| *** felipemonteiro__ has joined #openstack-infra | 13:55 | |
| fungi | as long as it continues to also work on our oldest supported distros (being ubuntu 14.04 and centos 7 at this point i think?) we're probably not opposed to switching up what network backend newer distros use for consistency's sake | 13:58 |
| mnaser | clarkb, kashyap: it would be nice to catch up and get started on some of the discussion we had re: nested kvm | 14:00 |
| * kashyap waves | 14:01 | |
| frickler | dirk: seems you managed to look at that tumbleweed node yesterday. do you still need it? | 14:02 |
| *** jcoufal has joined #openstack-infra | 14:03 | |
| *** shardy_ has quit IRC | 14:03 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:03 | |
| *** jcoufal_ has quit IRC | 14:04 | |
| dirk | frickler: no thanks.. I have a review | 14:04 |
| *** shardy has joined #openstack-infra | 14:05 | |
| *** eernst has quit IRC | 14:06 | |
| *** felipemonteiro__ has quit IRC | 14:07 | |
| *** pcichy has joined #openstack-infra | 14:07 | |
| *** yamamoto has joined #openstack-infra | 14:08 | |
| *** hamzy has quit IRC | 14:09 | |
| *** links has quit IRC | 14:10 | |
| *** jcoufal_ has joined #openstack-infra | 14:12 | |
| *** pcichy has quit IRC | 14:12 | |
| *** pcichy has joined #openstack-infra | 14:12 | |
| *** yamamoto has quit IRC | 14:13 | |
| *** hongbin has joined #openstack-infra | 14:13 | |
| kashyap | mnaser: The action item was to get a mailing list started | 14:14 |
| *** jcoufal has quit IRC | 14:14 | |
| mnaser | kashyap: right, i just wanted to start the conversation again to see if we can get there, so maybe when clarkb is around again we might be able to discuss | 14:14 |
| mnaser | i know he'll be at ONS though so i'll be seeing him in person too | 14:14 |
| kashyap | mnaser: What is "ONS"? | 14:15 |
| *** kiennt26_ has joined #openstack-infra | 14:15 | |
| mnaser | kashyap: sorry, should have clarified, open networking summit | 14:15 |
| *** dtantsur|brb is now known as dtantsur | 14:15 | |
| *** dklyle has joined #openstack-infra | 14:15 | |
| kashyap | (Yeah, it's still quite early for Clark.) | 14:15 |
| mnaser | https://events.linuxfoundation.org/events/open-networking-summit-north-america-2018/ | 14:15 |
| kashyap | Ah, I see. | 14:15 |
| *** jcoufal_ has quit IRC | 14:17 | |
| *** jcoufal has joined #openstack-infra | 14:18 | |
| *** germs_ has joined #openstack-infra | 14:18 | |
| *** germs has quit IRC | 14:20 | |
| *** alexchad_ has quit IRC | 14:22 | |
| *** cshastri has quit IRC | 14:23 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** jcoufal has quit IRC | 14:24 | |
| *** r-daneel has joined #openstack-infra | 14:27 | |
| prometheanfire | fungi: I think a good switchover for ubuntu will be 18.04 | 14:27 |
| *** yamamoto has quit IRC | 14:28 | |
| *** jcoufal has joined #openstack-infra | 14:30 | |
| fungi | makes sense | 14:30 |
| *** alexchadin has joined #openstack-infra | 14:31 | |
| fungi | on debian i'm still using either ifupdown on servers or networkmangler (via nmcli) on clients. i probably should figure out networkd at some point | 14:31 |
| tosky | fungi: but networkmanager is not deprecated | 14:32 |
| fungi | sure, but if networkd is "the future" i wouldn't mind gaining some familiarity with it | 14:32 |
| *** njohnston_ is now known as njohnston | 14:33 | |
| tosky | not sure if it's the absolute future | 14:33 |
| tosky | but let's see | 14:33 |
| *** jcoufal has quit IRC | 14:34 | |
| frickler | andreas_s: I'm seeing a couple of very late failures from zKVM CI, like for patches merged up to a week ago. just making sure you are aware of that. see e.g. https://review.openstack.org/553105 | 14:36 |
| *** sree has joined #openstack-infra | 14:37 | |
| andreas_s | frickler: thanks for noting - The CI is down at the moment (waiting for a fix in a third party library). Let me stop zuul that it stops voting | 14:37 |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** jcoufal has joined #openstack-infra | 14:39 | |
| *** hamzy has joined #openstack-infra | 14:40 | |
| andreas_s | ok, zuul-server is stopped now. It should stop voting now until the issues are fixed | 14:40 |
| *** jcoufal_ has joined #openstack-infra | 14:42 | |
| frickler | corvus: re zuul autohold, I found this traceback in zuul/debug.log, seems to be broken somehow http://paste.openstack.org/show/708967/ | 14:42 |
| *** mhayden has quit IRC | 14:43 | |
| *** yamamoto has quit IRC | 14:43 | |
| *** mhayden has joined #openstack-infra | 14:43 | |
| *** jcoufal__ has joined #openstack-infra | 14:44 | |
| *** jcoufal has quit IRC | 14:44 | |
| *** iyamahat has joined #openstack-infra | 14:45 | |
| Shrews | frickler: i think that's fixed, just needs a scheduler restart | 14:45 |
| dmsimard | infra-root: When troubleshooting the openstack ansible inventory issues on puppetmaster.o.o yesterday, I found out that we have thousands of leaked nodepool images. I'll work on cleaning that up today. | 14:45 |
| pabelanger | dmsimard: I want to say mordred had a script for shade to do that, last time | 14:46 |
| fungi | dmsimard: any commonality there? specific provider? specific builder? specific image>? | 14:46 |
| Shrews | dmsimard: or any nodepool metadata on them? | 14:46 |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| smcginnis | I have a zuul variable/post_test_hook question. | 14:46 |
| smcginnis | In python-brick-cinderclient-ext we had functional test failures because the post_test_hook was hard coded to chown to a "jenkins" owner. | 14:47 |
| *** jcoufal_ has quit IRC | 14:47 | |
| frickler | Shrews: oh, do you have a pointer to the fix? seems strange that it did seem to work for a different job yesterday | 14:47 |
| smcginnis | Lookin at other cases, it looks like other's post_test_hooks use the $user variable. | 14:47 |
| smcginnis | So I tried switching to that, but it doesn't appear to be defined. | 14:47 |
| Shrews | frickler: maybe https://review.openstack.org/547302 ? | 14:47 |
| smcginnis | Is there something else I need to do to pick that up? | 14:47 |
| dmsimard | fungi: I only looked at the openstackjenkins-rax DFW cloud (it was getting late), I'm not sure if there's a particular pattern yet. The list of images: https://paste.fedoraproject.org/paste/fS395oQcgGnQLstbVusZ6g (or in a more readable format: https://paste.fedoraproject.org/paste/YE~N562IeIraANwn3lNKcw/raw ) | 14:48 |
| dmsimard | Some images go back as far as March 8th according to the timestamps | 14:48 |
| dmsimard | Shrews: ^ | 14:48 |
| *** eernst has joined #openstack-infra | 14:48 | |
| dmsimard | pabelanger: It's not really that complicated to clean up, what I did in RDO before was to look at the nodepool images that were currently "current" (or "live") from nodepool's perspective and delete everything that was not those images | 14:49 |
| Shrews | oh, images, not nodes. nm | 14:49 |
| *** jcoufal has joined #openstack-infra | 14:49 | |
| *** jcoufal__ has quit IRC | 14:49 | |
| frickler | Shrews: corvus: oh, doesn't autohold work for POST_FAILURE at all? if build.result != "FAILURE": return in L1045 https://review.openstack.org/#/c/547302/2/zuul/scheduler.py | 14:49 |
| dmsimard | pabelanger: so it's like a simple loop ? openstack image list -f value -c ID |egrep -v "current_image|another_current_image" |xargs -L1 openstack image delete | 14:50 |
| Shrews | frickler: https://review.openstack.org/554995 | 14:50 |
| *** florianf has quit IRC | 14:51 | |
| *** florianf has joined #openstack-infra | 14:51 | |
| frickler | Shrews: I like how you come up with a new review for each of my questions ;) | 14:51 |
| pabelanger | dmsimard: should work, done that before but might take a while as you are loading CLI tools. mordred wrote a simple python script to do shade directly. | 14:51 |
| dmsimard | pabelanger: I can replace the xargs -L1 to xargs -P10 -L1 :P | 14:52 |
| frickler | but o.k., so we definitively need a scheduler restart for what I need | 14:52 |
| *** felipemonteiro__ has joined #openstack-infra | 14:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add autohold debug info https://review.openstack.org/555327 | 14:53 |
| corvus | frickler: oh, it was a post failure? i didn't know that. but yes, that's the reason it didn't work. | 14:54 |
| *** jcoufal has quit IRC | 14:54 | |
| corvus | frickler: anyway, there's a bunch of new debug info that probably would have helped me figure that out. :) | 14:54 |
| frickler | corvus: yes, failure when stackviz runs on py3 in post, seems related to new images built with new dib release | 14:55 |
| *** VW has joined #openstack-infra | 14:55 | |
| *** felipemonteiro_ has quit IRC | 14:57 | |
| *** ramishra has quit IRC | 14:57 | |
| *** Swami has joined #openstack-infra | 14:57 | |
| *** bobh has quit IRC | 14:57 | |
| *** gyankum has quit IRC | 14:58 | |
| *** alexchadin has quit IRC | 14:58 | |
| corvus | frickler: then yeah, we'll need a scheduler restart for that. | 14:59 |
| *** sree has quit IRC | 15:00 | |
| *** sree has joined #openstack-infra | 15:01 | |
| *** tpsilva has joined #openstack-infra | 15:03 | |
| *** agopi_out has joined #openstack-infra | 15:03 | |
| *** agopi_out is now known as agopi | 15:05 | |
| *** sree has quit IRC | 15:05 | |
| *** sree has joined #openstack-infra | 15:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit_configure flag to review01.o.o https://review.openstack.org/555334 | 15:08 |
| mordred | pabelanger: what am I doing? | 15:09 |
| pabelanger | mordred: at one point, you help me clean up stale images from nodepool. pretty sure it was a simple python app using shade to do that, vs the large grep / xargs command dmsimard is using | 15:10 |
| dmsimard | pabelanger: we don't have the same definition of large, I do bash one liners longer than that daily :P | 15:10 |
| *** jcoufal has joined #openstack-infra | 15:11 | |
| pabelanger | dmsimard: true, maybe slower? | 15:11 |
| pabelanger | since it is loading all of openstack client | 15:11 |
| *** sree has quit IRC | 15:12 | |
| dmsimard | maybe -- I mean, I'm still willing to look at what mordred may have written but I'm not that concerned about performance | 15:12 |
| *** hashar is now known as hasharAway | 15:13 | |
| mordred | dmsimard, pabelanger: whatever works for you ... I tend to use shade in a repl instead of openstackclient most times anyway - so I'm a bit of a nerd | 15:13 |
| *** dave-mccowan has joined #openstack-infra | 15:15 | |
| *** germs_ has quit IRC | 15:20 | |
| *** jcoufal has quit IRC | 15:20 | |
| clarkb | dmsimard: fungi aiui the problem was the combination of nodepool storing state in mysql db and trying to sync that with workers over gearman and the fairly complicated image creation process required by the glance api. This meant that if things went wrong after image creation during upload or syncing from swift you could leak images if you didn't fully unroll that process. And for the longest time | 15:20 |
| clarkb | the client tooling didn't unroll that and we just assumed well if it failed we delete it and move on which isn't always totally the case aiui | 15:20 |
| *** germs has joined #openstack-infra | 15:20 | |
| *** germs has quit IRC | 15:20 | |
| *** germs has joined #openstack-infra | 15:20 | |
| *** felipemonteiro__ has quit IRC | 15:21 | |
| clarkb | dmsimard: fungi I do think we've larlgely addressed this by having the builders talk to the db directly and manage state themselves (via zk) so it is harder for them to forget about things that need cleaning and we've also improved the state of unrolling in rools like shade | 15:21 |
| *** felipemonteiro__ has joined #openstack-infra | 15:21 | |
| stephenfin | AJaeger, mordred: What's this about not modifying '[testenv] install_command' to install upper constraints? https://review.openstack.org/528979 | 15:21 |
| clarkb | dmsimard: fungi I think the most interesting thing is not the oldest leaked image but the newest. Eg have changes to nodepool and shade made the situation better (and when I looked briefly yesterday it appeared the majority ofleaking stopped in septemberish which coincides with the switch to new nodepool builders I think | 15:22 |
| clarkb | dmsimard: as for using xargs -P10 the potential issue there is api limits but I doubt that osc will run fast enough to hit though even at -P 10 | 15:22 |
| *** armaan has joined #openstack-infra | 15:23 | |
| dmsimard | mordred: I've never thought of using shade interactively in a repl | 15:24 |
| dmsimard | mordred: that's interesting but I'm also not as intimate as you are with shade :p | 15:24 |
| dmsimard | clarkb: /me reads | 15:25 |
| dmsimard | clarkb: re: performance, I just meant that I don't care if the thing runs for 15 minutes instead of 5 -- it's a one time deal. If we realize that things are still leaking after that, the solution is not to make the cleanup faster... it's to figure out why nodepool leaks those and why can't it clean itself (like it would for VMs) | 15:27 |
| *** jcoufal has joined #openstack-infra | 15:27 | |
| *** kiennt26_ has quit IRC | 15:27 | |
| *** bobh has joined #openstack-infra | 15:28 | |
| clarkb | dmsimard: ++ I was just pointing out that in the past one of the problems with using xargs -PN with openstack APIs has been rate limits so you end up having to do multiple passes. But osc startup is not quick so mitigates that problem that way | 15:28 |
| clarkb | dmsimard: this was more of an issue using the old python-*clients directly | 15:28 |
| dmsimard | makes sense | 15:28 |
| *** Swami has quit IRC | 15:29 | |
| dmsimard | I think nodepool/shade is more hardcore on APIs than openstackclient :P | 15:29 |
| *** bobh has quit IRC | 15:30 | |
| *** bobh has joined #openstack-infra | 15:30 | |
| dmsimard | something I don't understand, though... is that yesterday, when I ran "alien-image-list".. it returned a puny amount of images considering what I was expecting to get back | 15:30 |
| dmsimard | so for some reason, nodepool might not see those at all or something | 15:30 |
| AJaeger | stephenfin: it does not work with installation of projects from git | 15:31 |
| AJaeger | stephenfin: mordred debugged this, hope he has more details | 15:31 |
| Shrews | dmsimard: nodepool ignores any images that do not have the nodepool_build_id in the image properties | 15:32 |
| dmsimard | mordred: btw I was seeing shade errors about floating IPs last night even with 1.27 for limestone, would like to see if you can make sense out of it | 15:32 |
| Shrews | dmsimard: alien == nodepool built, but missing info in zookeeper | 15:32 |
| clarkb | dmsimard: Shrews also the image api is paginated and if we aren't paging through properly the list will be short by default | 15:33 |
| dmsimard | Shrews: the images are bound to have nodepool metadata in them, they were uploaded by nodepool (see https://paste.fedoraproject.org/paste/fS395oQcgGnQLstbVusZ6g ) | 15:33 |
| Shrews | clarkb: true | 15:33 |
| *** jcoufal_ has joined #openstack-infra | 15:33 | |
| dmsimard | clarkb: the openstack ansible inventory properly paginates (through shade I assume) so that should be okay | 15:33 |
| dmsimard | Shrews: I can do an image show on one of those.. hang on | 15:33 |
| Shrews | dmsimard: i very seriously doubt nodepool uploaded the "Windows Server 2008 R2 SP1" image | 15:34 |
| Shrews | like, super srsly :) | 15:34 |
| dmsimard | super srsly, look at the 3200 other nodepool images :P | 15:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit_configure flag to review01.o.o https://review.openstack.org/555334 | 15:35 |
| *** ykarel has joined #openstack-infra | 15:35 | |
| *** jcoufal has quit IRC | 15:35 | |
| clarkb | but ya like I said I think zk and shade improvements have improved the state tracking and cleanup of failed builds | 15:35 |
| clarkb | if we have "modern" leaked images (which we didn't seem to have many of when I briefly looked) we should debug further but majority seem to be from older code that didn't grok glance as well | 15:36 |
| dmsimard | oh wow does rax overwrite nodepool properties ? http://paste.openstack.org/raw/708976/ | 15:36 |
| boden | mordred AJaeger hi, I recently sent a response to the ML regarding devstack impacts with tox_install/constraints http://lists.openstack.org/pipermail/openstack-dev/2018-March/128648.html looking for some advise on how to proceed | 15:37 |
| dmsimard | clarkb: those images are since the whereabouts of march 8th 2018, we're not looking at ancient prehistoric images | 15:37 |
| dmsimard | oh wait, maybe not | 15:38 |
| dmsimard | this one I pasted goes back to october | 15:38 |
| dmsimard | but one I looked was march 8th | 15:38 |
| dmsimard | ¯\_(ツ)_/¯ | 15:38 |
| Shrews | dmsimard: that'd be a problem if they do overwrite it | 15:39 |
| *** jamesmcarthur has joined #openstack-infra | 15:39 | |
| dmsimard | Shrews: If I look at the most recent image in my image list paste, (march 21st).. it doesn't have any nodepool properties, only all those rax metadata fields | 15:40 |
| dmsimard | let me look at a non-rax region .. | 15:40 |
| clarkb | dmsimard: if you look at centos7 the vast majority are pre october 2017 | 15:41 |
| pabelanger | dmsimard: clarkb: I want to say, when we made the cut to v3 builders, we likely didn't go back and properly clean up v2 images. Things were leaking pretty bad during that time, if I remember right. | 15:41 |
| clarkb | dmsimard: 5 are newer than that | 15:41 |
| clarkb | which is my point that we've likely largely fixed the problem | 15:41 |
| clarkb | we just never went back around and cleaned up the leaked images | 15:41 |
| dmsimard | Shrews: I don't see any leaks in ovh and they have the proper nodepool metadata | 15:41 |
| clarkb | pabelanger: yes also shade got a few improvements around this leaking then too (to better unroll failures) | 15:42 |
| dmsimard | clarkb: the issue is probably related to the fact that rax deletes the image properties/metadata that nodepool sets and puts their own instead | 15:42 |
| dmsimard | OVH doesn't do that, for example | 15:42 |
| dmsimard | clarkb: see RAX: http://paste.openstack.org/raw/708976/ vs OVH: http://paste.openstack.org/raw/708979/ | 15:42 |
| *** pcaruana has joined #openstack-infra | 15:42 | |
| pabelanger | clarkb: yah, we had some issue with rax VHDs too. Where if the image failed to uploaded, we didn't properly delete | 15:42 |
| clarkb | dmsimard: that may be why they don't show in the alien list today. But when we looked into the problems last year it was two fold of not being able to check state from the thing doing the deleting making it easier to "forget" and shade simply not unrolling on failures. We've since addressed both issues | 15:43 |
| clarkb | anyways looking at that list I don't see many "modern" leaks which makes me think we probably have largely addressed the problem. if you find evidence contrary to that we should investigate more | 15:43 |
| Shrews | dmsimard: can you check the most recent upload (say xenial) in rax? | 15:43 |
| dmsimard | Shrews: the most recent in rax also has stripped metadata | 15:44 |
| *** pcaruana has quit IRC | 15:44 | |
| Shrews | neat | 15:44 |
| openstackgerrit | Merged openstack-infra/zuul master: Add autohold debug info https://review.openstack.org/555327 | 15:44 |
| dmsimard | Shrews: from yesterday: http://paste.openstack.org/raw/708980/ | 15:44 |
| dmsimard | clarkb: We should probably let rax know that it's not okay to do that ? | 15:44 |
| dmsimard | clarkb: I mean, they can add properties if they want but deleting all the properties that users put is kinda meh | 15:44 |
| clarkb | dmsimard: they also use the non open source glance task for importing images from swift using an undocumented api | 15:45 |
| clarkb | I'm not sure we'll get very far on that but we can try | 15:45 |
| dmsimard | Otherwise, could nodepool somehow make sure it's properties are on there on a semi regular basis ? | 15:46 |
| dmsimard | Anyway, I'll start cleaning after lunch | 15:46 |
| dmsimard | but so far only rax has huge amounts of leaked things | 15:46 |
| Shrews | so alien-image-list will just never work on rax. i don't think that affects any other functionality | 15:47 |
| *** yamamoto has joined #openstack-infra | 15:47 | |
| clarkb | dmsimard: ya rax's particularly complicate upload process made it more susceptible to this aiui | 15:48 |
| clarkb | dmsimard: there are far more steps that you have to account for and any one of them failing would require unrolling and we just didn't do it well for a long time (since been addressed I think) | 15:48 |
| *** pcaruana has joined #openstack-infra | 15:48 | |
| dmsimard | yup, I'll keep an eye on it but I'm almost positive things are still going to leak | 15:48 |
| *** garyk has quit IRC | 15:49 | |
| dmsimard | we'll see | 15:49 |
| pabelanger | we should see if we can reproduce it just using shade with simple upload script | 15:49 |
| pabelanger | then work to fix it | 15:49 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: add query for tempest-full-py3 post failure bug 1758054 https://review.openstack.org/555302 | 15:50 |
| openstack | bug 1758054 in OpenStack-Gate "stackviz failing on py3 causing tempest-full-py3 POST_FAILURES" [Critical,New] https://launchpad.net/bugs/1758054 | 15:50 |
| clarkb | mnaser: kashyap yes! we should follow up on that. Probably next step is coming up with a neutral domain for ml hosting then adding the ml to our ml server | 15:51 |
| dmsimard | pabelanger: the issue is likely with the nodepool properties being overwritten by rax, I'm not sure what we can do to fix that other than 1) a dirty cron to delete leaked images 2) have nodepool query it's current image uuids and make sure it's properties are still on there | 15:51 |
| clarkb | fungi: ^^ may have opinions on how to go about that (having administered dns in the past) | 15:51 |
| clarkb | corvus too having just set one up for zuul-ci | 15:51 |
| kashyap | clarkb: Yep, I recall that much — about a neutral location; and a reasonable name (yay, bike-shedding) | 15:51 |
| clarkb | should we go ahead and use the new NS servers and go pay someone $10 for a .org? | 15:52 |
| *** ldnunes has quit IRC | 15:52 | |
| fungi | clarkb: kashyap: mnaser: i'm lacking context (probably owing to me only lightly skimming scrollback this morning due to other responsibilities vying for my time) | 15:53 |
| *** yamamoto has quit IRC | 15:53 | |
| kashyap | fungi: Context for you is surely missing | 15:53 |
| kashyap | (As only Clark, MNaser & myself were present in the Cloud W G room on one of the PTG days.) | 15:54 |
| clarkb | fungi: corvus sorry background here is that at PTG we (infra, devs (kashyap), and clouds (mnaser and public cloud wg)) how to better handle issues like $feature doesn't work in your cloud and we want to help debug it | 15:54 |
| clarkb | fungi: corvus the outcome of that seemed to be a neutral communication platform like a mailing list not under @openstack.org would be reasonable | 15:54 |
| fungi | oh, interesting | 15:54 |
| fungi | these are providers running openstack though? | 15:55 |
| clarkb | fungi: corvus the real gap we are trying to address here is the devs that can help debug are not cloud users or customers so can't go through normal support channels | 15:55 |
| * fungi is probably still missing some of the nuance which makes public openstack providers not want to speak publicly on an openstack.org mailing list) | 15:55 | |
| clarkb | fungi: today its because libvirt devs or whoever are likely to not want to openstack beacuse thats openstack | 15:56 |
| mordred | clarkb: gotcha - it's a place for devs and clouds to talk - not just openstack devs | 15:56 |
| clarkb | mordred: yesw | 15:56 |
| fungi | oh! so it's about getting devs for openstack dependencies outside our community to subscribe to a mailing list | 15:56 |
| fungi | is there not already such a mailing list somewhere out there on the 'net? | 15:57 |
| kashyap | mordred: Yeah, the devs can be Kernel, Sea Bios, KVM, QEMU, libvirt... | 15:57 |
| mordred | clarkb: I think that makes the 'neutral list' part make more sense - although to me there woudlnt' be any difference between a new list on a mailman with an o.o domain and one with a different domain | 15:57 |
| mordred | since it would be a new list and I'd have to deal with a new list | 15:57 |
| clarkb | fungi: apparently not | 15:57 |
| fungi | makes me wonder if we've really looked | 15:57 |
| mordred | but if it's a blocker to someone else, fine by me :) | 15:57 |
| clarkb | fungi: red hat has a cloud sig but for similar reasons thats all red hat specific | 15:57 |
| fungi | or just assuming there isn't an appropriate list already we can convince those people to join for better collaboration | 15:58 |
| clarkb | fungi: the brief looking I've done turned up the red hat cloud sig thing which if you use red hat cloud products/packages is great but otherwise its not very independent? | 15:58 |
| fungi | i mean, there are people who engage in discussion on the nanog ml even when they're neither north american nor network operators | 15:58 |
| clarkb | but double checking on that a bit more is probably a reasonable idea too | 15:58 |
| *** garyk has joined #openstack-infra | 15:59 | |
| mnaser | I think the idea was that we wanted something to extend past OpenStack world but I’m indifferent | 15:59 |
| clarkb | but at the end of the day the issue is I as a user file a bug with cloud. Cloud says whatever to me and the issue dies there. We need some way to bridge together the developers of various pieces of technology with the cloud operators so that we can debug problems more effectively rather than just (effectively) ignoring the user | 15:59 |
| mnaser | One of the things was the whole idea of some sort of privacy in terms of being able to start a conversation then take it off list | 15:59 |
| fungi | this is mostly the obligatory pushback on the yet-another-ml theme. if there are existing lists related to this topic nobody is using for it today, creating another one may have the opposite of the intended effect | 16:00 |
| pabelanger | infra-root: I'd like to bring review01.o.o online today, using out 2 step approach for review-dev01.o.o: https://review.openstack.org/555334 That will allow us to email out the new IP address to ML | 16:00 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: grenade: wait for service logs before matching against patterns https://review.openstack.org/493987 | 16:00 |
| * kashyap bbiab | 16:00 | |
| clarkb | fungi: ya. I'm not aware of any such thing today. But I've also likely barely skimmed for such things using search engine foo | 16:01 |
| pabelanger | clarkb: so, testing of review-dev01.o.o looks good, I haven't found any issues with gerrit so far. Do we want to open it up to other people to poke at? | 16:01 |
| clarkb | pabelanger: one thing to keep in mind with that switch that I must've figured out in my sleep is that we won't be able to puppet production gerrit during that time right? | 16:02 |
| clarkb | pabelanger: or at least the puppet-gerrit bits of puppet won't run | 16:02 |
| smcginnis | I asked a while back, but I think it was bad timing. Anyone know what needs to be done to pick up the $user variable in a python-brick-cinderclient-ext post test hook script? | 16:02 |
| *** ykarel has quit IRC | 16:02 | |
| clarkb | smcginnis: can you link to where it is used? | 16:02 |
| smcginnis | Or should I just switch the hardcoding from "jenkins" to "zuul" for the account name. | 16:02 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for OVB testenv failures https://review.openstack.org/502117 | 16:02 |
| smcginnis | clarkb: One moment... | 16:02 |
| clarkb | smcginnis: if it is used with tox then tox is likely filtering out the $USER env var | 16:02 |
| fungi | clarkb: kashyap: mnaser: cool. i'm not opposed on principle, but would hate for us to have done insufficient research and then create a new discussion forum which competes with an existing appropriate one we simply didn't know about. if we can find one we should convince the interested parties to go there instead | 16:03 |
| smcginnis | clarkb: https://review.openstack.org/#/c/555311/1/brick_cinderclient_ext/tests/functional/hooks/post_test_hook.sh | 16:03 |
| mordred | smcginnis: this is a hook script for an oldstyle devstack-gate job? | 16:03 |
| pabelanger | clarkb: no, review.o.o should still be able to run puppet, I left that node alone in site.pp, I copypasta into a new node for review01.o.o, which has gerrit disabled | 16:03 |
| smcginnis | mordred: It is failing in a legacy- job. | 16:03 |
| mordred | yah. | 16:03 |
| mordred | smcginnis: I'd just hardcode the jenkins->zuul switch in that | 16:03 |
| clarkb | pabelanger: ah so controlled by site.pp with different node defs? | 16:03 |
| *** udesale has quit IRC | 16:03 | |
| pabelanger | clarkb: right | 16:03 |
| *** armaan has quit IRC | 16:03 | |
| smcginnis | mordred: OK, that works for me. I would hope the chances of us making a big change like that any time soon is small. ;) | 16:04 |
| clarkb | smcginnis: mordred $USER shoudl work though. But that script uses $owner | 16:04 |
| pabelanger | clarkb: I did it that way for the reason you just listed | 16:04 |
| *** hemna_ has joined #openstack-infra | 16:04 | |
| *** ldnunes has joined #openstack-infra | 16:04 | |
| *** armaan has joined #openstack-infra | 16:04 | |
| clarkb | pabelanger: perfect | 16:04 |
| *** harlowja has joined #openstack-infra | 16:04 | |
| *** VW_ has joined #openstack-infra | 16:04 | |
| *** armaan has quit IRC | 16:04 | |
| smcginnis | clarkb: Ah, OK. Other scripts I saw had it lower case, so I figured it was something else. | 16:04 |
| kashyap | fungi: I agree on having to do sufficient research | 16:05 |
| mordred | smcginnis: well - what I'd REALLY do is replace that job with a nice new one that uses devstack-functional as a base job which knows how to run functional tox envs | 16:05 |
| *** felipemonteiro_ has joined #openstack-infra | 16:05 | |
| smcginnis | mordred: An examples I can take a look at? | 16:05 |
| kashyap | (I just don't know of any existing such forum.) | 16:05 |
| smcginnis | s/An/Any/ | 16:05 |
| mordred | smcginnis: yup! | 16:05 |
| * smcginnis always enjoys a good paint by numbers | 16:06 | |
| dhellmann | can I get another review on https://review.openstack.org/555034 so folks can approve the patches adding the lower-constraints jobs, please? | 16:06 |
| *** VW has quit IRC | 16:07 | |
| mordred | smcginnis: http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/.zuul.yaml#n42 <-- there's an example of how we're doing tox-based functional tests in openstacksdk | 16:07 |
| mordred | smcginnis: http://git.openstack.org/cgit/openstack/python-glanceclient/tree/.zuul.yaml is a similar example from glanceclient | 16:08 |
| mordred | smcginnis: I'm about to afk for about 45 minutes - but I can help you if you want to go down that path | 16:08 |
| clarkb | fungi: any ideas on how to better search for such a thing? Liek I said I basically hit a road block at the cloud sig | 16:08 |
| clarkb | kashyap: ^ | 16:08 |
| * kashyap reads scroll | 16:09 | |
| fungi | clarkb: lots and lots and lots of keyword searching is the only thing i can think of | 16:09 |
| kashyap | (A bit slow, as I am on a call.) | 16:09 |
| *** felipemonteiro__ has quit IRC | 16:09 | |
| kashyap | clarkb: Yeah, that's also my blocker — I don't know where to even start there. | 16:09 |
| mordred | smcginnis: hoenstly - the glanceclient file is a better copy-pasta source | 16:09 |
| *** harlowja has quit IRC | 16:09 | |
| fungi | clarkb: like, take an existing issue you're aware about and search as if you're trying to figure out the best place to discuss it so as to get representative input from relevant parties, and see what you come up with (if anything) | 16:10 |
| smcginnis | mordred: Thanks! | 16:10 |
| *** claudiub has joined #openstack-infra | 16:10 | |
| kashyap | fungi: That's a nice way to approach. I'll describe an example scenario: | 16:10 |
| fungi | clarkb: or at least somewhere with some of the relevant parties already represented, that you'd feel comfortable suggesting the other relevant parties join the discussion there | 16:10 |
| *** garyk has quit IRC | 16:11 | |
| kashyap | _If_ we are running level-2 guest in the upstream CI, and the level-1 guest (which is what you have access to) crashes | 16:11 |
| fungi | and repeat with a few scenarios if you have that many and see if the results are somewhat consistent | 16:11 |
| kashyap | KVM devs might want have access to stuff like CPUID, and other crash dumps, etc | 16:12 |
| kashyap | Yeah, probably should write an Etherpad with example sceanrios as fungi asked | 16:12 |
| clarkb | kashyap: ya then we can apply them to search engines and see what we get | 16:12 |
| * kashyap nods | 16:13 | |
| *** eharney has quit IRC | 16:14 | |
| fungi | this way when we inevitably end up duplicating effort with some other group running a similar ml, we can at least refer back to our failure to find theirs ;) | 16:15 |
| *** yamamoto has joined #openstack-infra | 16:17 | |
| clarkb | kashyap: is that something you want to start on? (I think you probably have more of this paged in than anyone else) | 16:17 |
| kashyap | Sure, not eager to duplicate efforts :-) | 16:17 |
| *** udesale has joined #openstack-infra | 16:18 | |
| kashyap | clarkb: Does this Etherpad name (haven't hit 'return' yet) look OK: etherpad.openstack.org/Collab-Clouds-and-Contribs | 16:18 |
| clarkb | kashyap: sure | 16:18 |
| fungi | wfm | 16:18 |
| kashyap | Poor name, better names welcome | 16:18 |
| * kashyap just spells it out: etherpad.openstack.org/Collab-Clouds-and-Contributors | 16:18 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: add query for tempest-full-py3 post failure bug 1758054 https://review.openstack.org/555302 | 16:18 |
| openstack | bug 1758054 in OpenStack-Gate "stackviz failing on py3 causing tempest-full-py3 POST_FAILURES" [Critical,New] https://launchpad.net/bugs/1758054 | 16:18 |
| * fungi puts out his "no bikeshedding" sign | 16:18 | |
| clarkb | ya name seems fine to me | 16:18 |
| kashyap | Heh, /me puts the paint back in the garage | 16:18 |
| kashyap | Okay, we can start typing: https://etherpad.openstack.org/p/Collab-Clouds-and-Contributors | 16:19 |
| kashyap | :-) | 16:19 |
| * fungi needs an actual sign which says that, to bring to conferences | 16:19 | |
| *** andreas_s has quit IRC | 16:19 | |
| corvus | fungi: should it sit on the table, or do you want to hang it on the door? | 16:19 |
| corvus | or both? | 16:19 |
| fungi | probably goes on the door | 16:19 |
| corvus | or maybe an A-frame sandwich board you can set outside if the door doesn't have a hook | 16:19 |
| fungi | "no bikeshedding beyond this point" | 16:20 |
| corvus | fungi: i'm thinking helvetica | 16:20 |
| corvus | that's always good for public information notices | 16:20 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Remove python-brick-cinderclient-ext functional test https://review.openstack.org/555365 | 16:20 |
| * fungi sighs | 16:20 | |
| mnaser | so question | 16:20 |
| mnaser | is there some sort of kvm-users ML | 16:20 |
| mnaser | maybe if we just convinced operators to get to kvm-users to resolve those type of things | 16:21 |
| mnaser | looks like there is general kvm | 16:21 |
| mnaser | but its a very very very busy ML from what it seems like, with all dev work going there | 16:22 |
| *** yamamoto has quit IRC | 16:22 | |
| mnaser | libvirt-users doesn't seem like it's the right place either because it's not a libvirt issue | 16:22 |
| clarkb | mnaser: ya I think thats the problem with using tech specific forums | 16:22 |
| clarkb | it is a very diverse set of problems and not needing to rebootstrap in each specific forum would be nice | 16:23 |
| *** udesale has quit IRC | 16:23 | |
| fungi | so far my image of the proposed forum is "something like nanog but for server virtualization technologies" | 16:23 |
| mnaser | the thing that stayed with me since that convo was | 16:23 |
| clarkb | chances are once some initial communication happens things will shift over to tech specific forums beacuse that is where you'll debug nested virt for example. But users and clouds and maybe even devs won't know necesarily that is where they need to go without having some intiail conversations | 16:24 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: add openstack-tox-lower-constraints https://review.openstack.org/555034 | 16:24 |
| mnaser | i like what fungi proposes but i wonder if it will ever actually grow outside openstack world | 16:24 |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Explicitly read setup.cfg as utf-8 on Python 3 https://review.openstack.org/555366 | 16:24 |
| mnaser | also on a lighter note, we were just offered to be in a cover story feature ... for "The 10 Best Performing AWS Solution Providers, 2018" | 16:25 |
| mnaser | the things people come up with these days.. | 16:25 |
| dmsimard | mnaser: you do AWS things ? Congrats anyway :p | 16:26 |
| fungi | aws is just another word for clouds, right? | 16:26 |
| mnaser | (also, recruiters contacting me to "sit down" at ONS because they saw from my tweets that i'll be there) | 16:26 |
| mnaser | s/recruiters/aws recruiters/ | 16:26 |
| * mnaser shrugs | 16:26 | |
| smcginnis | Did I see some stackviz issue being discussed earlier? I see a failure on one of our jobs. | 16:27 |
| mnaser | anyways, back to topic at hand, i'd love for this idea to stretch beyond openstack but im not sure it'll be able to get that visiblity. i think openstack-operators might be a good place to start | 16:27 |
| *** masayukig has quit IRC | 16:29 | |
| *** bobh has quit IRC | 16:29 | |
| fungi | smcginnis: see more detailed discussion in #openstack-qa | 16:29 |
| smcginnis | fungi: Ah, maybe that's where I saw it. Thanks | 16:30 |
| *** manjeets has joined #openstack-infra | 16:30 | |
| fungi | smcginnis: best summary we have is that it seems like with our latest images from today, the tempest python3 jobs are now using python3 instead of python2 to install stackviz on xenial | 16:30 |
| smcginnis | And stackviz is py2 only? | 16:31 |
| fungi | which is probably the right thing anyway, but means that we've spotted a data quality or data handling problem causing stackviz to break under real use | 16:31 |
| fungi | i don't think stackviz is intended to be python2-only | 16:31 |
| timothyb89 | fungi, smcginnis: it is not supposed to be py2 only, no | 16:31 |
| clarkb | fungi: that is weird that the image would affect that? pip and python are segregated by version aren't ther? | 16:31 |
| fungi | it just may lack some protections around handling non-utf-8 content in subunit streams | 16:32 |
| clarkb | pip3 and python3 vs pip and python | 16:32 |
| prometheanfire | mordred: AJaeger: what do you think of converting the config drive network data into a iproute2 script? Then it'd be truly universal... | 16:32 |
| *** yamamoto has joined #openstack-infra | 16:32 | |
| smcginnis | Should we hold off on rechecking things until this is sorted? | 16:32 |
| fungi | clarkb: i agree, i'm not convinced it's the image but it started around the same hour today's images went into production and we couldn't immediately spot any related changes merging to involved repos around the same timeframe | 16:32 |
| fungi | smcginnis: i believe frickler has a change proposed to tempest to make the stackviz task more resilient | 16:33 |
| prometheanfire | mordred: AJaeger blame cloudnull for that suggestion | 16:33 |
| clarkb | prometheanfire: I don't think we want to be in that business of competing with all the distros and now networkd | 16:33 |
| cloudnull | wat?! | 16:33 |
| prometheanfire | cloudnull: your fault | 16:33 |
| smcginnis | fungi: OK, I'll hold tight for a bit. | 16:33 |
| clarkb | re networkd my only concern with it is it requires using resolvd | 16:33 |
| fungi | smcginnis: bug 1758054 | 16:33 |
| clarkb | which I'm not a fan of because of dns over dbus | 16:33 |
| openstack | bug 1758054 in OpenStack-Gate "stackviz failing on py3 causing tempest-full-py3 POST_FAILURES" [Critical,New] https://launchpad.net/bugs/1758054 | 16:33 |
| *** andreas_s has joined #openstack-infra | 16:33 | |
| clarkb | which has had multiple cves against it as its kind of a silly idea | 16:33 |
| prometheanfire | does networkd require resolvd? | 16:33 |
| cloudnull | clarkb: its not required. some of the functionality in networkd may not be available if resolved is not present but it works otherwise | 16:34 |
| clarkb | prometheanfire: if you set dhcp to configure nameservers it does | 16:34 |
| clarkb | which for the vast majority of users is likely to be the case | 16:34 |
| prometheanfire | bah | 16:34 |
| fungi | smcginnis: proposed fix has been approved but not yet merged: https://review.openstack.org/555314 | 16:34 |
| cloudnull | https://review.openstack.org/#/c/552721 -- we're converting our testing in osa to use networkd across ubuntu cent and suse | 16:34 |
| fungi | smcginnis: er, s/fix/workaround/ | 16:34 |
| cloudnull | and suse does not have resolved available | 16:35 |
| smcginnis | fungi: Just saw that. I'll watch for that to land before rechecking anything. | 16:35 |
| clarkb | cloudnull: my reading of the docs is that if you don't have resolvd enabled then you have to configure resolv.conf outside of networkd | 16:35 |
| clarkb | cloudnull: it is doable | 16:35 |
| cloudnull | yes that's true | 16:35 |
| smcginnis | Hmm, it's got a whole bunch of red though. | 16:35 |
| clarkb | and infra would do it (we already ignore dhcp specified nameservers) | 16:36 |
| kashyap | mnaser: clarkb: I wrote some words here, feel free to correct / add: https://etherpad.openstack.org/p/Collab-Clouds-and-Contributors | 16:36 |
| *** rlandy is now known as rlandy|biab | 16:36 | |
| *** masuberu has quit IRC | 16:36 | |
| *** eharney has joined #openstack-infra | 16:36 | |
| *** yamamoto has quit IRC | 16:36 | |
| clarkb | prometheanfire: I do think having that as an option in glean to simplify cross distro differences is an excellent feature | 16:37 |
| clarkb | prometheanfire: might also be the answer to ipv6 support on all the distros "use networkd" then we don't have to solve it different each time | 16:38 |
| prometheanfire | clarkb: dns is already handed off in config drive | 16:38 |
| *** jcoufal_ has quit IRC | 16:38 | |
| clarkb | prometheanfire: not always. | 16:38 |
| clarkb | prometheanfire: config drive could say "use dhcp" and leave it at that iaui | 16:38 |
| prometheanfire | well, it should be lol | 16:38 |
| *** sshnaidm is now known as sshnaidm|off | 16:38 | |
| pabelanger | I thought we disabled DNS update in glean from config-drive | 16:39 |
| prometheanfire | systemd still calls out to dhclient or dhcpcd for dhcp though, and those write directly to resolve.conf iirc | 16:39 |
| clarkb | pabelanger: not by default | 16:39 |
| pabelanger | on our images | 16:39 |
| clarkb | prometheanfire: its a flag iirc | 16:39 |
| clarkb | er pabelanger ^ | 16:39 |
| pabelanger | right, but I thought we added the flag to our images | 16:39 |
| pabelanger | checking | 16:39 |
| clarkb | pabelanger: ya we do. But we aren't the only users of glean | 16:39 |
| pabelanger | okay, ya | 16:39 |
| prometheanfire | so, what would be prefered for writing out the network info, networkd or iproute2 script? | 16:40 |
| clarkb | prometheanfire: fwiw I'm actually of the opinion that the opposite is true. I really wish clouds would stop trying to force staticnetwork config on all the instances forcing us to have tools like glean in the first place. I would much appreciate it if they all just agreed to use dhcp and then everything could just work with default images | 16:40 |
| clarkb | prometheanfire: networkd probably. an iproute2 script is going to have to tie into the init system anyways and at that point you are reinventing networkd | 16:41 |
| prometheanfire | clarkb: I'd be happier with that too | 16:41 |
| *** iyamahat has quit IRC | 16:41 | |
| prometheanfire | as far ask networkd, ack | 16:41 |
| prometheanfire | s/ask/as | 16:41 |
| clarkb | then have users that have special exceptional needs like vlan support or link aggregation opt in to some statically provided network configuration details and then only they have to worry about special tooling to realize that configuration | 16:42 |
| prometheanfire | if it's part of the data handed off by config-drive networkd should be able to set up the interface | 16:44 |
| prometheanfire | whatever it is | 16:44 |
| prometheanfire | I'll probably get started working on this tonight | 16:44 |
| *** yamahata has quit IRC | 16:45 | |
| *** yamamoto has joined #openstack-infra | 16:47 | |
| *** andreas_s has quit IRC | 16:47 | |
| tosky | infra (and zuul) people, I have a problem with some future jobs for sahara, and I have few choices, but it depends also on you | 16:49 |
| tosky | in order to proper test some of the sahara plugins, I would like to add few additional multinode jobs | 16:50 |
| tosky | now, resources are limited, so I would like to limit the impact of those jobs | 16:50 |
| tosky | we have one of those multinode jobs in the experimental queue right now, but if we add more, running all of them in one shot would be too much | 16:51 |
| tosky | so I was wondering if it's possible to define more queues, so that we can do "check sahara-<something>" | 16:51 |
| tosky | a more generic solution could be enabling users to do "check <job name>" | 16:52 |
| *** yamamoto has quit IRC | 16:52 | |
| tosky | separate - but related - issue: apart from the usage in specific reviews, it would be nice to add them as periodic job, which we could do now | 16:52 |
| tosky | but how could we make sure that it does not overload the gates? Are periodic jobs executed when the gates are unloaded? | 16:53 |
| tosky | (which means Saturday morning? :) | 16:53 |
| fungi | there is the "experimental" pipeline, and "check experimental" will run the jobs defined therein | 16:54 |
| *** caphrim007 has joined #openstack-infra | 16:54 | |
| fungi | i'm assuming you're not asking about having the ability to get zuul to leave verify +1 when only (re)running a subset of jobs | 16:54 |
| *** wolverineav has joined #openstack-infra | 16:54 | |
| *** wolverineav has quit IRC | 16:54 | |
| *** wolverineav has joined #openstack-infra | 16:55 | |
| fungi | periodic jobs get lowest priority scheduling, so if there are gate or check jobs in the queue they'll be prioritized for node assignment purposes | 16:55 |
| fungi | instead of any pending periodic jobs | 16:55 |
| fungi | note that the post pipeline also shares the same precedence (priority) as periodic | 16:57 |
| tosky | about the experimental queue: yes, we have one job there, but it does not scale if we add more | 16:57 |
| *** iyamahat has joined #openstack-infra | 16:57 | |
| tosky | and no, I'm not asking about getting targeted +1 | 16:57 |
| fungi | so the idea is that you want to be able to request non-voting runs of individual jobs on demand? | 16:57 |
| tosky | yes, exactly | 16:57 |
| fungi | seems like that would result in an explosion of new pipeline definitions | 16:58 |
| tosky | if it's implemented with pipelines, probably | 16:58 |
| tosky | but if experimental would recognize something like "check experimental mynicejob", maybe not | 16:59 |
| fungi | trigger definitions are pipeline-specific (at least at the moment), so the only way i'm aware of to trigger specific jobs would be by defining separate pipelines for them | 16:59 |
| tosky | but I don't know the internals to know if this could be too complicated to implement | 16:59 |
| *** gyee has joined #openstack-infra | 17:00 | |
| tosky | well, but then there is no need for more pipelines: it could be either the existing experimental or a special experimental-single-job queue with a special behavior | 17:00 |
| fungi | as a feature request, it's probably more on topic for #zuul. i know it's come up before and we've had reasons not to support it, but i don't necessarily know if those reasons still hold true today so can't hurt to discuss as a future option | 17:00 |
| corvus | tosky: it's not "trivial" but it's probably not too hard; it's come up a few times, and i think we can consider it, but we don't have a plan for it right now. iow, we won't be able to accomplish it by next week so you may not want to wait on it :) | 17:01 |
| clarkb | one workaround has been to only enable the job on the pipeline in the change | 17:01 |
| clarkb | then remove it when you are happy with the results | 17:01 |
| tosky | no, of course I'm not asking about it *RIGHT NOW!!!!1!111!!" of course :) | 17:01 |
| tosky | I would start by creating the periodic jobs | 17:01 |
| tosky | yes, that's possible, it's just feel a workaround and new reviews | 17:02 |
| *** yamamoto has joined #openstack-infra | 17:02 | |
| tosky | is it now a good time for reraising it on #zuul ? Are those answers enough to try to open a story (do you use storyboard, right?) | 17:02 |
| *** garyk has joined #openstack-infra | 17:02 | |
| corvus | tosky: yeah, you can go ahead and file a story. most of us are pretty focused on getting 3.0 out the door right now, so it might be best to raise it after the release (hopefully RSN) | 17:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add publish-zuul-docs job https://review.openstack.org/555377 | 17:05 |
| tosky | corvus: ack, thanks; I will write a story to not forget | 17:06 |
| *** yamamoto has quit IRC | 17:06 | |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thanks! | 17:16 |
| *** yamahata has joined #openstack-infra | 17:16 | |
| *** yamamoto has joined #openstack-infra | 17:17 | |
| corvus | pabelanger, clarkb: can you review https://review.openstack.org/555377 for zuul docs publishing? | 17:18 |
| *** diablo_rojo has quit IRC | 17:19 | |
| *** yamamoto has quit IRC | 17:22 | |
| *** garyk has quit IRC | 17:22 | |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| *** trown is now known as trown|lunch | 17:24 | |
| *** diablo_rojo has joined #openstack-infra | 17:24 | |
| *** panda is now known as panda|off | 17:26 | |
| *** danpawlik has quit IRC | 17:27 | |
| tosky | fungi: just one additional question about periodic jobs: apart from the priority, how frequently are they scheduled? | 17:27 |
| *** sambetts is now known as sambetts|afk | 17:27 | |
| clarkb | tosky: daily (around 0600UTC iirc) | 17:28 |
| tosky | ack, thanks | 17:29 |
| *** samuel has joined #openstack-infra | 17:29 | |
| fungi | tosky: definition lives at https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n158 | 17:31 |
| fungi | but yes, time: '0 6 * * *' | 17:31 |
| *** jpich has quit IRC | 17:31 | |
| tosky | and what happens if for some reason a periodic job stays in the queue for 24 hours, until the next run? Does the new run cancels the old one? | 17:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-zuul-docs job https://review.openstack.org/555377 | 17:31 |
| fungi | nope | 17:32 |
| fungi | they both get run | 17:32 |
| *** yamamoto has joined #openstack-infra | 17:32 | |
| fungi | generally with nodes assigned in the order in which they were enqueued | 17:32 |
| tosky | couldn't this produce an excessive load if for some reason the gates are already overloaded? | 17:32 |
| fungi | the times where we're under enough load to go more than 24 hours without running periodic jobs are fairly few | 17:32 |
| clarkb | and gearman priority largely addresses that | 17:33 |
| fungi | and due to having precedence: low they don't hold up check or gate jobs | 17:33 |
| fungi | right, that | 17:33 |
| *** florianf has quit IRC | 17:33 | |
| tosky | so basically in the worst case, in one or two days, when the load lowers, the situation usually stabilize...? | 17:34 |
| fungi | basically, for each new node assignment, precedence: high gets filled first, then precedence: normal and finally precedence: low if available | 17:34 |
| fungi | tosky: right | 17:34 |
| pabelanger | fungi: clarkb: corvus: could I also get a few reviews on https://review.openstack.org/555334/ review01.o.o server so we can send out the new IP for upcoming rebuild | 17:34 |
| *** garyk has joined #openstack-infra | 17:35 | |
| *** derekh has quit IRC | 17:35 | |
| *** yamamoto has quit IRC | 17:37 | |
| *** samuel is now known as samuelbeau | 17:38 | |
| *** e0ne has quit IRC | 17:39 | |
| *** esberglu_ has joined #openstack-infra | 17:39 | |
| *** rlandy|biab is now known as rlandy | 17:40 | |
| *** lucasagomes is now known as lucas-afk | 17:40 | |
| *** esberglu has quit IRC | 17:41 | |
| *** tosky has quit IRC | 17:44 | |
| smcginnis | Sorry to ask, but we have some python-brick-cinderclient-ext patches blocked until we sort out the current legacy functional gate failures. If someone has the time, this is a simple removal - https://review.openstack.org/#/c/555365/1 | 17:44 |
| pabelanger | looks like we have a few nodes in ovh-bhs1 / ovh-gra1 stuck deleting | 17:46 |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| *** tosky has joined #openstack-infra | 17:48 | |
| pabelanger | Shrews: if you have a moment, I'm curious why we have a 22day old ready node (unlocked) in nodepool. 0002739297 is the ID | 17:48 |
| pabelanger | mordred: can we also clean up your held node? | 17:48 |
| pabelanger | mordred osc job breakage | 17:48 |
| timothyb89 | fungi, smcginnis: this patch should fix the underlying stackviz bug: https://review.openstack.org/555388 | 17:49 |
| Shrews | pabelanger: is there anything in our logs for that node still? | 17:49 |
| smcginnis | timothyb89: Excellent, thanks! | 17:49 |
| *** Anticimex has joined #openstack-infra | 17:50 | |
| pabelanger | Shrews: possible, let me look quickly | 17:50 |
| *** pblaho has quit IRC | 17:50 | |
| pabelanger | Shrews: doesn't look like it | 17:51 |
| *** yamamoto has quit IRC | 17:52 | |
| pabelanger | Shrews: oh, my mistake it is locked | 17:52 |
| pabelanger | I think I know what might be the issue | 17:52 |
| *** danpawlik has joined #openstack-infra | 17:52 | |
| pabelanger | I haven't had a change to fix a bug where zuul will lock a noderequest, then a reload of config happens and the job is removed. we are left with a lost noderequest lock | 17:53 |
| pabelanger | I'll work on that today to fix | 17:53 |
| *** felipemonteiro has joined #openstack-infra | 17:56 | |
| *** danpawlik has quit IRC | 17:57 | |
| mordred | pabelanger: yes please | 17:57 |
| mordred | pabelanger: (that is, yes, please do delete that node if you're there already) | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Publish zuul docs to zuul-ci.org https://review.openstack.org/555395 | 17:58 |
| pabelanger | mordred: ack, will delete now | 17:58 |
| *** felipemonteiro_ has quit IRC | 17:59 | |
| dmsimard | pabelanger: btw I'm receiving cron emails from review-dev about what seems to be an error | 18:00 |
| pabelanger | dmsimard: ack, looking | 18:00 |
| *** yankcrime has quit IRC | 18:00 | |
| dmsimard | /usr/local/lib/python2.7/dist-packages/paramiko/client.py:779: UserWarning: Unknown ssh-rsa host key for [review-dev01.openstack.org]:29418: 03cb1cc7df724fe7d53cc06a8d9cef5c key.get_name(), hostname, hexlify(key.get_fingerprint()), | 18:00 |
| pabelanger | hmm, wonder if something isn't puppeted | 18:00 |
| *** jpena is now known as jpena|off | 18:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/storyboard master: Switch to using stestr for tests https://review.openstack.org/555397 | 18:01 |
| *** yamamoto has joined #openstack-infra | 18:02 | |
| pabelanger | dmsimard: Oh, right ssh host keys | 18:05 |
| pabelanger | they would have changed | 18:05 |
| pabelanger | clarkb: dmsimard: in fact, I think we want to use our old host keys right? otherwise, everybody is going to have to accept new keys? How did we do this in the past when replacing gerrit | 18:05 |
| *** yamamoto has quit IRC | 18:06 | |
| *** yamamoto has joined #openstack-infra | 18:06 | |
| dmsimard | does gerrit present it's own ssh host key ? or is that straight out of sshd ? | 18:06 |
| *** yamamoto has quit IRC | 18:06 | |
| fungi | pabelanger: pretty sure the host keys we care about are in the cinder volume | 18:06 |
| clarkb | dmsimard: it has its own ssh host keys | 18:06 |
| pabelanger | oh, right | 18:06 |
| clarkb | fungi: ya they are in review_site/etc iirc | 18:07 |
| fungi | the host key puppet cares about is entirely separate | 18:07 |
| pabelanger | hmm, let me debug more | 18:07 |
| fungi | pabelanger: my guess is that 1. old review.dev.o.o hasn't been deleted yet (or is still in our ansible inventory cache), 2. the review-dev.o.o dns name is now an alias for review-dev01.o.o, so 3. puppetmaster is trying to connect to _both_ review-dev01.o.o and review-dev.o.o not realizing they're the same system | 18:08 |
| fungi | when connecting to review-dev.o.o it gets the ssh host key for the new review-dev01.o.o and complains because that's not what it's expecting (understandably, since it's not the same server it thinks it is) | 18:09 |
| pabelanger | fungi: this is actually /usr/local/bin/track-upstream that is failing on review-dev01.o.o | 18:09 |
| pabelanger | trying to see which key it is using | 18:09 |
| fungi | where's the error getting logged? | 18:10 |
| pabelanger | emails to infra-root via crontab | 18:10 |
| pabelanger | see dmsimard above | 18:10 |
| clarkb | for ^ chacnes are the known hosts for that user didn't get copied as part of the volume move | 18:10 |
| *** camunoz has joined #openstack-infra | 18:10 | |
| clarkb | so while the key didn't change we didn't preserve the ssh known hosts file for the user doing ssh to the gerrit | 18:10 |
| pabelanger | yah, I think something isn't puppeted | 18:11 |
| fungi | oh, i wonder if /usr/local/bin/track-upstream ran before the cinder volume was copied over and initially cached a regenerated ssh host key for the api | 18:11 |
| fungi | or, right, ~/.ssh_known_hosts is somewhere other than the cinder volume so didn't get preserved | 18:12 |
| *** boris_42_ has joined #openstack-infra | 18:12 | |
| *** rpittau has quit IRC | 18:12 | |
| pabelanger | yah, it is running as root, so I guess it was always manually accepted the key | 18:13 |
| pabelanger | I mean, why doesn't it run as gerrit user? | 18:13 |
| pabelanger | guess file permissions on disk | 18:13 |
| clarkb | possibly because cron + puppet + not root can be a pita | 18:14 |
| clarkb | or I guess its exec maybe cron is fine | 18:14 |
| pabelanger | let me manually accept key and see if that fixes it. if so, I'll update puppet to manage it | 18:14 |
| *** harlowja has joined #openstack-infra | 18:15 | |
| *** harlowja_ has joined #openstack-infra | 18:17 | |
| pabelanger | k, that was it | 18:17 |
| pabelanger | dmsimard: thanks! fixing | 18:17 |
| clarkb | dmsimard: how goes limestone mirror? are we ready to start building images and boot nodes yet? | 18:19 |
| *** harlowja has quit IRC | 18:19 | |
| dmsimard | clarkb: In a meeting right now but I need to troubleshoot a shade floating IP issue | 18:19 |
| dmsimard | clarkb: I can't launch the mirror anymore :( | 18:19 |
| dmsimard | I pinged mordred about it earlier this morning -- mordred if you have a chance, would love to look at ^ | 18:20 |
| mordred | dmsimard: ok. I can take a look | 18:20 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-brick-cinderclient-ext-dsvm-functional https://review.openstack.org/555401 | 18:20 |
| mordred | dmsimard: is what you're trying to run and having issues with in scrollback anywhere? (I'd like to start from where you already are) | 18:20 |
| *** dave-mccowan has quit IRC | 18:20 | |
| mordred | smcginnis: woot. the devstack-functional tests worked out for you? | 18:21 |
| *** samuelbeau has quit IRC | 18:22 | |
| smcginnis | mordred: It would appear so. Thanks for the pointers there. That wasn't bad at all. | 18:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove python-brick-cinderclient-ext functional test https://review.openstack.org/555365 | 18:23 |
| clarkb | mordred: smcginnis related to that I kinda wish more of the functional tests didn't actualyl require devstack | 18:24 |
| clarkb | nova has done a good job with this aiui | 18:24 |
| *** trown|lunch is now known as trown | 18:25 | |
| smcginnis | clarkb: I guess it comes down to what each team decides "functional test" means. | 18:25 |
| smcginnis | clarkb: We had way too much discussion about it at a midcycle once. | 18:25 |
| *** danpawlik has joined #openstack-infra | 18:25 | |
| smcginnis | I prefer a lighter approach too, but I don't think everyone was convinced. | 18:26 |
| clarkb | ya I mean ship has sailed and neutron sort of led the way for a lot of it which is why we see less of the nova setup I think | 18:26 |
| clarkb | (but neutron functional testing is more complicated than running a full cloud for a tempest run as a result)_ | 18:26 |
| openstackgerrit | Doug Hellmann proposed openstack/diskimage-builder master: add lower-constraints job https://review.openstack.org/555412 | 18:27 |
| dmsimard | mordred: I don't think I've pasted the error, let me go ahead and get a fresh one | 18:27 |
| *** danpawlik has quit IRC | 18:30 | |
| *** trown has quit IRC | 18:31 | |
| *** ralonsoh has quit IRC | 18:31 | |
| AJaeger | pabelanger, dirk, is tumbleweed working fine so that we can add it as gating test to openstack-zuul-jobs - see https://review.openstack.org/#/c/546844/ ? I'm happy to +A but wanted to double check here first | 18:31 |
| *** tesseract has quit IRC | 18:32 | |
| johnsom | Can someone take a quick look at https://review.openstack.org/#/c/554960/ where xgerman is trying to set an artifact destination? Thanks. | 18:32 |
| fungi | clarkb: you mentioned the `black` utility yesterday... looks like pypa may end up using it to format the code for the warehouse service: https://github.com/pypa/warehouse/pull/3367 | 18:32 |
| *** VW_ has quit IRC | 18:32 | |
| *** VW has joined #openstack-infra | 18:32 | |
| AJaeger | johnsom: LGTM | 18:33 |
| johnsom | Thank you! Just wasn't sure that was the right path or not. | 18:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit ssh hostkey to root for manage-projects https://review.openstack.org/555414 | 18:34 |
| pabelanger | AJaeger: wfm, +A when ready | 18:35 |
| AJaeger | johnsom: it looked good - but I still suggest to double check after next run ;) | 18:35 |
| *** germs has quit IRC | 18:36 | |
| *** germs has joined #openstack-infra | 18:37 | |
| *** germs has quit IRC | 18:37 | |
| *** germs has joined #openstack-infra | 18:37 | |
| *** diablo_rojo has quit IRC | 18:38 | |
| *** germs has quit IRC | 18:39 | |
| *** germs has joined #openstack-infra | 18:39 | |
| *** germs has quit IRC | 18:39 | |
| *** germs has joined #openstack-infra | 18:39 | |
| *** lpetrut_ has quit IRC | 18:39 | |
| *** diablo_rojo has joined #openstack-infra | 18:40 | |
| openstackgerrit | Merged openstack-infra/zuul master: Publish zuul docs to zuul-ci.org https://review.openstack.org/555395 | 18:41 |
| mordred | clarkb, fungi: ooh. that actually formats things how I like them to be formatted - with one exception that I can totally learn to live with | 18:41 |
| clarkb | mordred: my big issue with it is it requires python 3.6 which is installed basically nowhere currently | 18:42 |
| clarkb | I do like the focus is on making code review easier | 18:42 |
| *** felipemonteiro has quit IRC | 18:42 | |
| AJaeger | mordred: how do you want to continue with https://review.openstack.org/531825 ? To remove all tox_install.sh copies, we need this up since we have projects that e.g. need nova. is the release team on board with publishing everything to pypi? | 18:43 |
| *** felipemonteiro has joined #openstack-infra | 18:43 | |
| *** Krenair_ is now known as Krenair | 18:45 | |
| *** Krenair has joined #openstack-infra | 18:45 | |
| *** dprince has quit IRC | 18:46 | |
| mordred | smcginnis, dhellmann: ^^ ? | 18:47 |
| mordred | AJaeger: I guess question #1 is - is the release team happy with that idea ... alternately, we could just add nova to the publish list | 18:48 |
| dhellmann | did I miss a mailing list thread on that? | 18:48 |
| smcginnis | Seems like we do need wider input on that. | 18:48 |
| mordred | dhellmann: no. or, well, yes, you did, but not because you didn't read it, more because it doesn't exist | 18:49 |
| smcginnis | I'm fine doing that, I just don't want to make any unilateral decisions for everyone. | 18:49 |
| mordred | kk. want me to kick one off? | 18:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add opensuse-tumbleweed testing to ozj https://review.openstack.org/546844 | 18:49 |
| dhellmann | I personally can no longer context switch to deep thought issues related to infra that only come up in hallways or irc. I apologize for my shortcomings there, but I'm going to just keep asking for ML threads a lot of the time. | 18:49 |
| smcginnis | mordred: I think that would be good. | 18:49 |
| dhellmann | and I'm sure this was discussed at the ptg, but I wasn't able to attend those sessions because of the schedule :/ | 18:49 |
| mordred | dhellmann: no apologies necessary ... I have a hard time dealing with mailing lists in any form, so I pretty much always have to be reminded/asked | 18:50 |
| dhellmann | I'd be OK with a spec or something, too -- anything with paragraphs instead of timestamps | 18:50 |
| mordred | dhellmann: it wasn't really - at least not in dublin - tonyb and I talked about it some in denver - it's mostly been IRC conversations thus-far | 18:50 |
| mordred | dhellmann: nod. | 18:50 |
| dmsimard | mordred: sorry, got stuck in back to back meetings. Here's the floating ip trace: http://paste.openstack.org/show/708991/ | 18:50 |
| dhellmann | ok. I realized I was potentially mischaracterizing discussions I wasn't aware of, so I was trying to acknowledge that | 18:50 |
| mordred | dhellmann: we're providing an excellent two-human example of how there is no single communication mechanism that works well for everyhone | 18:51 |
| dhellmann | indeed | 18:51 |
| * fungi would settle for a communication mechanism that works well for even most hones | 18:51 | |
| mordred | dhellmann: I'll kick off a mailing list thread | 18:51 |
| dhellmann | mordred : thank you | 18:51 |
| *** eernst has quit IRC | 18:51 | |
| dhellmann | sorry for being pedantic | 18:51 |
| mordred | fungi: :) | 18:51 |
| corvus | clarkb: can you please re-review https://review.openstack.org/549252 -- i added the requested tests | 18:51 |
| clarkb | corvus: ya | 18:52 |
| mordred | dmsimard: awesome, thanks. | 18:52 |
| corvus | clarkb: sorry, meant https://review.openstack.org/522054 | 18:52 |
| mordred | dmsimard: ah - wow, you weren't kidding - neutron just isn't available there is it? | 18:52 |
| *** amoralej is now known as amoralej|off | 18:53 | |
| mordred | I'm honestly not sure if _nova_attach_ip_to_server has ever been used in anger | 18:53 |
| fungi | dhellmann: mordred: fwiw, i would have found an ml thread around it useful as well. apparently changes removing tox-install.sh from projects entirely ran afoul of forgetting we have local devs who need to do things similar to our ci jobs but we don't (yet) have a tool to get the bits needed for them to be able to directly use the tox-siblings ansible role locally | 18:53 |
| mordred | as we have never had a cloud with floating ips but without neutron api | 18:53 |
| dhellmann | fungi : yes, I did see the thread about that being broken | 18:53 |
| openstackgerrit | Merged openstack-infra/release-tools master: import the old onboarding presentation https://review.openstack.org/545058 | 18:54 |
| clarkb | corvus: funny enough because of rebasing the diffs in the first have the content from the second :) | 18:54 |
| mordred | dmsimard, clarkb: which human is our limestone friend again? | 18:54 |
| fungi | mordred: you want logan- | 18:54 |
| mordred | fungi: thanks! | 18:54 |
| logan- | o/ | 18:54 |
| pabelanger | linestone config is public too | 18:54 |
| pabelanger | limestone* | 18:55 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone-specific network configuration to *clouds.yaml https://review.openstack.org/555423 | 18:55 |
| dmsimard | mordred: ^ btw | 18:55 |
| logan- | https://github.com/continuouspw/openstack-ci-cloud/ <- cloud config | 18:55 |
| dmsimard | mordred: I had been testing with my own clouds.yaml with that patch incorporated in it | 18:55 |
| dmsimard | mordred, logan-: it's kind of weird too because launch-node had been working before | 18:56 |
| dmsimard | I mean, once we got the clouds.yaml config down | 18:56 |
| clarkb | corvus: lgtm thanks | 18:56 |
| dmsimard | so I'm wondering if the network changes logan- did had something to do with it | 18:56 |
| mordred | dmsimard, logan-: so - I see floating_ip_source: nova - but I see neutron in the catalog | 18:56 |
| mordred | is there something blocking the neutron floating ip api calls? | 18:57 |
| mordred | (the nova ones are inherently racey and impossible to use without leaking) | 18:57 |
| mordred | c._network_client.get('/floatingips.json') works | 18:58 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable systemd profile for Gentoo https://review.openstack.org/552952 | 18:58 |
| logan- | dmsimard was seeing floating ip pool errors from neutron | 18:59 |
| *** danpawlik has joined #openstack-infra | 18:59 | |
| mordred | nod. but it's not something you're intending for people to not use? | 18:59 |
| logan- | correct | 18:59 |
| mordred | cool. | 18:59 |
| dmsimard | mordred: oh, yeah -- if you do "openstack floating ip pool list" it's going to complain about compute v2 something something | 18:59 |
| mordred | ah - that's becuase openstack floating ip pool list is a command to work with floating ip pools in the nova api | 19:00 |
| dmsimard | let me see if I put neutron instead of nova in the floating ip source what happens | 19:00 |
| mordred | well- that clouds.yaml value is not read by openstackclient | 19:01 |
| mordred | what you want is 'openstack floating ip list' | 19:01 |
| dmsimard | yeah openstack floating ip list works | 19:01 |
| dmsimard | but floating ip pool list is different, it tells you what pools to pick from (especially if the default one isn't set which tends to happen) | 19:01 |
| mordred | right. | 19:01 |
| mordred | but pools aren't actually a thing | 19:01 |
| mordred | networks are the thing | 19:01 |
| dmsimard | ¯\_(ツ)_/¯ | 19:01 |
| dmsimard | yeah ofc | 19:01 |
| mordred | pools is the nova network concept | 19:02 |
| mordred | which is now dead | 19:02 |
| *** r-daneel_ has joined #openstack-infra | 19:02 | |
| * mordred apologizes for the existence of the word pool in shade, fwiw, he didn't know anybetter 4 years ago | 19:02 | |
| *** r-daneel has quit IRC | 19:02 | |
| *** r-daneel_ is now known as r-daneel | 19:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit ssh hostkey to root for manage-projects https://review.openstack.org/555414 | 19:03 |
| mordred | dmsimard: all that said- we should probably at least provide a better error message than what you got | 19:03 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-brick-cinderclient-ext-dsvm-functional https://review.openstack.org/555401 | 19:04 |
| dmsimard | mordred, logan-: it works after taking the nova floating ip source :/ | 19:04 |
| *** jamesmcarthur has quit IRC | 19:04 | |
| *** danpawlik has quit IRC | 19:04 | |
| mordred | \o/ | 19:04 |
| logan- | woot | 19:04 |
| mordred | that's much better than the other option | 19:04 |
| mordred | dmsimard: I know you're busy today - at some point it might be good to post-mortem with you on the pain you had in finding a correct config and the ways in which we can help mitigate that | 19:05 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone-specific network configuration to *clouds.yaml https://review.openstack.org/555423 | 19:05 |
| dmsimard | mordred: the idea behind me stepping up to do this was to take a stab at actually learning and documenting the process :) | 19:06 |
| mordred | dmsimard: ++ | 19:06 |
| dmsimard | sorry if it's been taking so long between the different issues and lack of availability | 19:06 |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| pabelanger | dmsimard: left comment | 19:07 |
| mordred | dmsimard: oh, golly - no need to be sorry! there are unfortunately many things to leran inthis space | 19:07 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone-specific network configuration to *clouds.yaml https://review.openstack.org/555423 | 19:07 |
| dmsimard | pabelanger: fixed, thanks | 19:07 |
| mordred | dmsimard, logan-: reading that config change tells me that Gateway Network is a network that VMs boot on to and it gives them a private ipv4 fixed ip and a public ipv6 fixed ip - and that Flat External is only used as a source for floating ips, yes? | 19:08 |
| dmsimard | mordred: Canadians are sorry a lot, I have this bookmarked for sharing :P http://www.cbc.ca/2017/sorry-can-we-talk-about-why-canadians-apologize-so-much-1.3939997 | 19:08 |
| jroll | where are the clouds.yaml files defined for infra's nodepool? | 19:08 |
| logan- | yessir | 19:08 |
| pabelanger | jroll: system-config | 19:08 |
| jroll | pabelanger: thanks | 19:09 |
| dmsimard | jroll: right now here: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb | 19:09 |
| clarkb | jroll: https://review.openstack.org/#/c/555423/ includes changes to them | 19:09 |
| mordred | jroll: modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb is one of them ... | 19:09 |
| clarkb | ha | 19:09 |
| jroll | :) | 19:09 |
| dmsimard | it's like we're all fighting to see who will answer jroll first | 19:09 |
| mordred | actually - those are both puppetmaster files | 19:09 |
| pabelanger | jroll: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb | 19:09 |
| * jroll has no problem with that | 19:09 | |
| jroll | thank you all | 19:09 |
| *** jamesmcarthur has joined #openstack-infra | 19:10 | |
| mordred | there it is | 19:10 |
| dmsimard | pabelanger is the technically correct person :D | 19:10 |
| *** Swami has joined #openstack-infra | 19:10 | |
| clarkb | and then the nodepool clouds.yaml will only know about gateway network and we won't put anything on flat external from there | 19:11 |
| pabelanger | heh, we still have infracloud there, guess I need to rebase some patches | 19:12 |
| dmsimard | yeah logan- mentioned nodepool will be v6 only | 19:12 |
| dmsimard | puppet is running on the mirror yay \o/ | 19:13 |
| *** yamamoto has quit IRC | 19:13 | |
| dmsimard | mordred: oh also I was wondering why does shade does a get on every image for the openstack inventory | 19:13 |
| dmsimard | cause, like, we have these 3k leaked images and it was really going through each of them | 19:14 |
| mordred | dmsimard: really? that's super-strange | 19:14 |
| dmsimard | it's only later that I realized there was a nodepool tenant in my clouds.yaml | 19:14 |
| dmsimard | mordred: yeah, I would see that with the --list --debug | 19:14 |
| pabelanger | woot | 19:14 |
| dmsimard | mordred: I probably have a paste.. hang on | 19:14 |
| mordred | dmsimard: the only time the inventory should do anything with images is to fill in details on servers | 19:15 |
| mordred | dmsimard: which we can (and should) disable in our config | 19:15 |
| dmsimard | mordred: paste.openstack.org/show/708378/ | 19:15 |
| dmsimard | not sure why there's no http in that link: http://paste.openstack.org/show/708378/ | 19:15 |
| Swanson | Getting post_failure on https://review.openstack.org/555393 This is a test fixture thing, right? | 19:15 |
| dmsimard | mordred: the inventory poking at the images is actually what tipped me on the leaked images | 19:16 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove the propose-update-requirements job https://review.openstack.org/555426 | 19:16 |
| mordred | dmsimard: well, one good thing I suppose :) | 19:16 |
| pabelanger | Swanson: http://logs.openstack.org/93/555393/2/check/openstack-tox-py27/22e6818/job-output.txt.gz#_2018-03-22_18_52_14_365067 | 19:16 |
| pabelanger | looks like test failure | 19:16 |
| dmsimard | Swanson: the failure is here: http://logs.openstack.org/93/555393/2/check/openstack-tox-py27/22e6818/ara/result/1d6371d1-12b9-45a6-850d-124f2592be21/ | 19:16 |
| mordred | dmsimard: AH - so - that's doing a single paginated list_images | 19:17 |
| mordred | (which makes sense if we have thousands of them) | 19:17 |
| pabelanger | and possible issue with playbooks, we shouldn't collect testr is test folder missing | 19:17 |
| dmsimard | mordred: not sure about single, it kept on going for a good 15 minutes before I interrupted it | 19:17 |
| pabelanger | http://logs.openstack.org/93/555393/2/check/openstack-tox-py27/22e6818/job-output.txt.gz#_2018-03-22_18_52_28_266947 | 19:17 |
| mordred | dmsimard: yah - well, what was probably happening is that it was doing a list_images for every server in that cloud :) | 19:17 |
| mordred | dmsimard: and we also don't have any caching turned on in that clouds.yaml | 19:18 |
| dmsimard | mordred: hardcore | 19:18 |
| dmsimard | mordred: so do we care about images ? | 19:18 |
| mordred | dmsimard: so - you know- it was just a lovely nested-loop | 19:18 |
| Swanson | Oh, shoot. Maybe I should push that fix, too. | 19:18 |
| mordred | dmsimard: no, not in that context - patch coming | 19:18 |
| dmsimard | mordred: awesome thanks <3 | 19:18 |
| *** zoli is now known as zoli|gone | 19:18 | |
| *** zoli|gone is now known as zoli | 19:18 | |
| dmsimard | (I guess you could consider this part of "post mortem") | 19:19 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove the propose-update-requirements job https://review.openstack.org/555426 | 19:20 |
| dmsimard | ok, puppet has finished running on the mirror and there's a failure at the very end but that's because launch-node.py tries an old thing with a new shade .. I could probably send a patch for that. | 19:22 |
| dmsimard | for the record: http://paste.openstack.org/raw/708993/ | 19:22 |
| pabelanger | oh, that is for dns | 19:23 |
| dmsimard | yeah | 19:23 |
| dmsimard | I'm sending a patch, easy fix | 19:23 |
| pabelanger | ack | 19:23 |
| mordred | dmsimard: ooh. yes to patch that! | 19:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Update launch-node's dns.py to work with modern shade https://review.openstack.org/555433 | 19:27 |
| dmsimard | mordred, pabelanger ^ should do it | 19:28 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Update launch-node's dns.py to work with modern shade https://review.openstack.org/555433 | 19:28 |
| mordred | dmsimard: honestly, just remove the try/except and do the cloud.list_servers() | 19:28 |
| efried | dhellmann: Is here where I would bug you about that ceilometer-powervm lower-constraints issue? | 19:29 |
| mordred | oh! wait | 19:29 |
| mordred | dmsimard: that's not going to work at all | 19:29 |
| dmsimard | mordred: is that going to work with 1.11 ? I just don't want to rabbit hole all the way down | 19:29 |
| dmsimard | why not ? | 19:29 |
| dhellmann | efried : sure | 19:29 |
| efried | esberglu_, edmondsw: ^ | 19:29 |
| mordred | dmsimard: it'll work with 1.11 - except it won't ... because we strip rest links dicts, so get_href() is gonna fail | 19:29 |
| efried | dhellmann: So what I'm failing to understand is how this is different from other tox jobs that use our tox.ini to pull in the ceilometer repo. | 19:30 |
| mordred | efried: ECONTEXT (trying to butt in) | 19:30 |
| mordred | dmsimard: I'll push up a followup for you | 19:30 |
| dmsimard | mordred: sure | 19:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add missing localhost delegation checks to some modules https://review.openstack.org/555435 | 19:30 |
| dhellmann | efried : I'm not sure I can answer that. I'm scripting several dozen of these lower-constraints changes | 19:30 |
| efried | dhellmann: Care to join us in #openstack-powervm so mordred doesn't get madder? | 19:30 |
| dhellmann | mordred : the lower-constraints job for ceilometer-powervm failed because it didn't install ceilometer (I think) | 19:31 |
| *** robled has quit IRC | 19:31 | |
| dhellmann | efried : he might have your answer, since I know he's working on some of this stuff | 19:31 |
| efried | Oh, mordred I thought you were getting huffy because I was butting into the existing conversation. | 19:31 |
| dhellmann | mordred : I have run into the same problem with several other repos. I don't know the current state of the art for specifying dependencies on other repos. | 19:31 |
| mordred | dhellmann: well - turns out it's related to the earlier conversation on which I was going to start a mailing list thread | 19:32 |
| efried | dhellmann, mordred: We get ceilometer for our other tox-ic stuff via https://review.openstack.org/#/c/555358/1/tox.ini@15 | 19:32 |
| dhellmann | mordred : I had a strong suspicion it was | 19:33 |
| *** danpawlik has joined #openstack-infra | 19:33 | |
| mordred | dhellmann: tl;dr - dependencies should be delcared in requirements.txt and should use normal pypi packages - we've actually had a long-standing policy against the use of git urls for requirements - which is enforced by the requirements repo | 19:33 |
| dhellmann | efried : I'm not sure how you're going to want to specify that for lower-constraints. You'd maybe want an older version of ceilometer? | 19:33 |
| dhellmann | mordred : so we need to get everyone publishing, then we need to update the tox settings | 19:34 |
| efried | mordred: Um, I'm pretty sure we want exactly the same version as ceilometer-powervm. | 19:34 |
| mordred | dhellmann: yes | 19:34 |
| dhellmann | and then do we have my dream situation where we stop testing against master of services in their plugins? or do we have some magic that pulls in master? | 19:34 |
| mordred | efried: what you don't want is modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb | 19:34 |
| mordred | dhellmann: we have some magic that CAN pull in master | 19:35 |
| mordred | dhellmann: we use it in python-openstackclient, for instance, to have the -tips jobs as well as the normal jobs | 19:35 |
| efried | I have no idea what that is. | 19:35 |
| mordred | efried: sorry - bad paste :) | 19:35 |
| mordred | efried: -egit+https://github.com/openstack/ceilometer#egg=ceilometer | 19:35 |
| efried | But okay, am I allowed to say `ceilometer` in my requirements.txt? | 19:35 |
| *** dhajare has joined #openstack-infra | 19:35 | |
| mordred | efried: you are- as soon as we start uploading it to pypi | 19:35 |
| efried | ah. Which is... not yet? | 19:35 |
| mordred | efried: not yet - I need to start a mailing list thread so we can get the buy-in we need real quick | 19:36 |
| efried | Gotcha. So for right now I'll just mark https://review.openstack.org/#/c/555358/ as waiting for that to happen. WFM. Thanks mordred dhellmann. | 19:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add missing localhost delegation checks to some modules https://review.openstack.org/555435 | 19:37 |
| mordred | dhellmann: the 'magic' is that in zuul if you list another project in your required-projects list, AND that project has a corresponding entry in your requirements list so that it gets installed normally ... then zuul will install the git version for that job - this also allows depends-on links to work properly | 19:37 |
| *** danpawlik has quit IRC | 19:37 | |
| mordred | dhellmann: the missing thigns are a) consensus about uploading servers to pypi b) a utility function to allow people to (correctly) do the same thing locally | 19:38 |
| *** patriciadomin has quit IRC | 19:38 | |
| dhellmann | the utility bit was the patch to pbr that I put a -1 on because I didn't recognize what it was for (see our earlier conversation about ML threads) | 19:39 |
| mordred | anyway - I'll send out a mailng list thread so we can keep unwinding this | 19:39 |
| *** _nick has joined #openstack-infra | 19:39 | |
| mordred | dhellmann: yah - I've also got a todo to respond to that | 19:39 |
| dhellmann | yeah. corvus pinged me on it and I said I would think about it more and I think I'm probably going to be OK with it once I have the full context. | 19:39 |
| dhellmann | I mean, I still hate it, but it is what it is and I think we have to have it | 19:40 |
| *** _nick is now known as yankcrime | 19:40 | |
| mordred | dhellmann: well - we could also make it its own utility and not have it in pbr - there is nothing about it that requires it be in the pbr repo | 19:41 |
| dhellmann | oh, I'm opposed to the whole thing, not just where it lives :-) | 19:41 |
| dhellmann | I think we should start being less tightly coupled. | 19:42 |
| mordred | dhellmann: so I think it's definitely worth discussing not only the utility, but where it lives | 19:42 |
| dhellmann | sure | 19:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add missing localhost delegation checks to some modules https://review.openstack.org/555435 | 19:42 |
| *** dhajare has quit IRC | 19:42 | |
| mordred | dhellmann: oh - yah - I mean, the utility to me is more to provide an easy-ish way of saying "I want to test that this patch I just made to keystoneuath locally does the thing I want in openstacksdk" - and for people who aren't me (cause I just do that by hand all the time by doing .tox/py35/bin/pip install -e ../keystoneuath) | 19:43 |
| mordred | dhellmann: I *don't* want to make a thing that's about turning out python deps all into git refs | 19:43 |
| *** samuelbeau has joined #openstack-infra | 19:44 | |
| *** esberglu_ is now known as esberglu | 19:44 | |
| dhellmann | mordred : ok. I suspect it's going to be leveraged for that more than I like but I do see the usefulness of what you're proposing. | 19:46 |
| *** r-daneel_ has joined #openstack-infra | 19:49 | |
| *** r-daneel has quit IRC | 19:50 | |
| *** r-daneel_ is now known as r-daneel | 19:50 | |
| mriedem | if someone hasn't done so yet, i'd like to formally request that 555314 goes to the top of the queue because nothing in the integrated-gate is going to pass until that merges | 19:54 |
| *** rlandy is now known as rlandy|brb | 19:54 | |
| clarkb | pabelanger: I've approved the gerrit configure change | 19:54 |
| clarkb | mriedem: that the stackviz fix? makes sense to me | 19:54 |
| mriedem | it's a workaround | 19:55 |
| mriedem | for now | 19:55 |
| *** trown has joined #openstack-infra | 19:55 | |
| clarkb | it failed a voting check job | 19:55 |
| clarkb | anyone look into if that was a valid failure yet? | 19:55 |
| mriedem | smcginnis did | 19:55 |
| mriedem | http://logs.openstack.org/14/555314/1/check/tempest-full-py3/1e90d86/job-output.txt.gz#_2018-03-22_16_15_03_178612 | 19:56 |
| timothyb89 | clarkb, mriedem: https://review.openstack.org/#/c/555388/ fixes the root cause | 19:56 |
| clarkb | timothyb89: oh maybe we focus on that one then | 19:56 |
| mriedem | that requires approval and a release and u-c change right? | 19:56 |
| clarkb | mriedem: just approval and an image rebuild I think | 19:56 |
| pabelanger | clarkb: woot, thanks! | 19:57 |
| clarkb | I don't have the +2s on that project though | 19:57 |
| clarkb | happy to work on getting the workaround in too | 19:57 |
| clarkb | smcginnis: given the failure on 555314 are you comfortable direct enqueing it into the gate and promoting it at this point? | 19:57 |
| fungi | clarkb: timothyb89: i think the challenge with 555388 is that we grab the sdist for it from tarballs.openstack.org and bake that into our images via the cache-devstack element, so would then need an image build to get the right one into devstack jobs? | 19:59 |
| clarkb | fungi: yes | 20:00 |
| fungi | oh, you did say image rebuild | 20:00 |
| pabelanger | fungi: clarkb: dmsimard: https://review.openstack.org/555414/ should fix our host key issue from this afternoon for root user on review.o.o / review-dev.o.o | 20:00 |
| clarkb | fungi: mriedem timothyb89 it wouldn't be a bad thing to get the workaround in then the revert of workaround will be self testing when we have images updated | 20:02 |
| clarkb | allows us to do that at a more measured pace so I'm fine working btoh changes at the same time | 20:02 |
| fungi | clarkb: i don't know that the workaround is entirely a workaround. we don't want other bugs in stackviz to start causing post_failure results in jobs using it, do we? | 20:02 |
| *** gfidente has quit IRC | 20:03 | |
| mriedem | you can revert the workaround, | 20:03 |
| *** eharney has quit IRC | 20:03 | |
| mriedem | once the fix lands | 20:03 |
| mriedem | and not hold up the rest of openstack from merging code today | 20:03 |
| clarkb | mriedem: right thats what I'm saying and it will be self testing | 20:03 |
| clarkb | fungi: as for that I guess it depends on how important the reports are | 20:03 |
| clarkb | fungi: eg are they important enough that not having them should be treated as a failure? | 20:03 |
| mriedem | i would say no muy importante | 20:03 |
| fungi | true, i was going on the assumption that failure to be able to generate a stackviz report shouldn't be considered a hard failure for the job | 20:04 |
| smcginnis | Agree | 20:04 |
| mriedem | mtreinish i might have a bot in his closet that dies, | 20:04 |
| mriedem | but that's no small loss | 20:04 |
| clarkb | I think for regular logs it most definitely would be a failure | 20:04 |
| mriedem | *big | 20:04 |
| clarkb | (because you wouldn't actually be able to verify success at that point) | 20:04 |
| *** danpawlik has joined #openstack-infra | 20:06 | |
| *** samuelbeau has quit IRC | 20:06 | |
| *** robled has joined #openstack-infra | 20:07 | |
| clarkb | fungi: looks like the workaround is basically just a "don't fail the job if this task fails" change | 20:07 |
| clarkb | so maybe we dont' ervert it is what we are saying and just check that stackviz works again once images update? | 20:08 |
| * clarkb goes to figure out how to enqueue it to the gate and promote it | 20:08 | |
| *** yamamoto has joined #openstack-infra | 20:09 | |
| fungi | right | 20:10 |
| clarkb | mriedem: smcginnis fungi promotion is done | 20:11 |
| mriedem | woot, thanks | 20:11 |
| *** danpawlik has quit IRC | 20:12 | |
| edmondsw | mordred when did that zuul magic start? We've noticed some weirdness in our stable/queens branch that I think that may have caused... local UT finding a policy.json file but not zuul because of a different path | 20:12 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable systemd profile for Gentoo https://review.openstack.org/552952 | 20:13 |
| *** yamamoto has quit IRC | 20:14 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add gerrit_configure flag to review01.o.o https://review.openstack.org/555334 | 20:15 |
| *** samuelbeau has joined #openstack-infra | 20:15 | |
| *** samuelbeau has joined #openstack-infra | 20:16 | |
| * clarkb realizes we should probably warn ceilometer of their upcoming rename? | 20:18 | |
| clarkb | In theory it isn't a surprise but its also been a long time sinc ewe have been able to do one of these | 20:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Add missing localhost delegation checks to some modules https://review.openstack.org/555435 | 20:20 |
| clarkb | I'll work on getting ready to restart executors for ^ | 20:25 |
| clarkb | puppet run starting in 4 minutes should pick that up which will update the executors about 10 minutes after it starts so expect restarts around then | 20:26 |
| *** rlandy|brb is now known as rlandy | 20:30 | |
| *** danpawlik has joined #openstack-infra | 20:39 | |
| *** boris_42_ has quit IRC | 20:41 | |
| *** danpawlik has quit IRC | 20:43 | |
| *** samuelbeau has quit IRC | 20:43 | |
| clarkb | 9141ae5 is installed on all 10 ze's I'm going to restart them now | 20:44 |
| clarkb | the stop has been issues just waiting for them to actually stop | 20:48 |
| *** trown is now known as trown|outtypewww | 20:50 | |
| *** trown|outtypewww has quit IRC | 20:50 | |
| clarkb | executors starting now | 20:54 |
| *** edmondsw has quit IRC | 20:56 | |
| clarkb | looks like jobs are starting up again | 20:56 |
| *** edmondsw has joined #openstack-infra | 20:56 | |
| *** Adri2000 has quit IRC | 20:58 | |
| *** edmondsw has quit IRC | 21:01 | |
| *** jamesmcarthur has quit IRC | 21:02 | |
| *** hamzy has quit IRC | 21:03 | |
| *** kgiusti has left #openstack-infra | 21:07 | |
| ianw | ... is it actually possible to get consumption of output of an execed subprocess correct in python3? i'm starting to think the answer is "no" | 21:07 |
| dansmith | clarkb: were you promoting the stackviz change to the front of the gate? | 21:08 |
| clarkb | dansmith: yes its alraedy been done | 21:08 |
| dansmith | clarkb: is it merged such that I should recheck or is it still in gate? | 21:09 |
| clarkb | dansmith: still in the gate | 21:09 |
| dansmith | ack thanks | 21:09 |
| dansmith | clarkb: I saw several more post_failures today, that's unrelated right? | 21:09 |
| clarkb | correct | 21:09 |
| clarkb | or at least there is more than one post failure issue :/ | 21:09 |
| dansmith | okay | 21:10 |
| *** yamamoto has joined #openstack-infra | 21:10 | |
| *** jcoufal has joined #openstack-infra | 21:11 | |
| dmsimard | pabelanger: commented on https://review.openstack.org/#/c/555414/ | 21:11 |
| dmsimard | ianw: consumption of output ? | 21:12 |
| dmsimard | ianw: you mean like... subprocess.check_output ? | 21:12 |
| clarkb | ianw: it is clunky | 21:13 |
| *** jamesmcarthur has joined #openstack-infra | 21:13 | |
| clarkb | iirc you have to be careful that you don't block on a read that won't emit any data until you write something | 21:13 |
| *** jcoufal has quit IRC | 21:13 | |
| *** agopi is now known as agopi|out | 21:14 | |
| *** danpawlik has joined #openstack-infra | 21:14 | |
| pabelanger | dmsimard: yah, good idea about ipaddress6, will push up update | 21:15 |
| dmsimard | logan-, infra-root: so http://mirror.regionone.limestone.openstack.org/ is a thing (if DNS has propagated on your end), I'll send the nodepool patches to get the ball rolling | 21:15 |
| pabelanger | yay | 21:15 |
| dmsimard | pabelanger: see also my comment about other potential known hosts that we'd be erasing | 21:15 |
| *** yamamoto has quit IRC | 21:16 | |
| pabelanger | dmsimard: yah, i plan on back up current and doing a diff, then adding missing contents to puppet | 21:16 |
| clarkb | dmsimard: mirror is up for me \o/ | 21:16 |
| clarkb | I've got to go figure out some packing for travel tomorrow though | 21:16 |
| dmsimard | pabelanger: the problem is the known_hosts is hashed on review.o.o right now so we don,t really know what they are | 21:16 |
| dmsimard | (also you're adding unhashed entries so I'm not sure how that works) | 21:17 |
| clarkb | dmsimard: ya debuntu do that by default | 21:17 |
| dmsimard | ssh-keyscan might be better | 21:17 |
| clarkb | tumbleweed doesn't and its like wait this is an option is so much nicer | 21:17 |
| dmsimard | clarkb: it's for security purposes | 21:17 |
| pabelanger | unhased should be fine, we do that for executors | 21:17 |
| dmsimard | pabelanger: yeah but I mean can you actually mix unhashed with hashed ? | 21:17 |
| pabelanger | will confirm | 21:18 |
| clarkb | dmsimard: ya it just seems minimally useful if I've been compromised already | 21:18 |
| pabelanger | but don't think it is an issue | 21:18 |
| *** agopi|out has quit IRC | 21:18 | |
| pabelanger | but, I'm more interested in why known_hosts has other hashes. possible root users of just manually added them over time | 21:19 |
| *** danpawlik has quit IRC | 21:19 | |
| *** salv-orlando has quit IRC | 21:19 | |
| *** gouthamr has quit IRC | 21:19 | |
| dmsimard | pabelanger: known_hosts last modified in november 2017 | 21:19 |
| *** salv-orlando has joined #openstack-infra | 21:19 | |
| pabelanger | yah, we'll know fast if we depend on the data, once we bring review01.o.o online, it will be empty | 21:20 |
| *** eernst has joined #openstack-infra | 21:20 | |
| clarkb | once we have images up and before we increase max servers it is often good to run a devstack + tempest job just to get a baseline idea of how well things will perform | 21:20 |
| *** rfolco|ruck is now known as rfolco|off | 21:20 | |
| clarkb | and sort out any bugs like the /32 ovh uses breaking nova net | 21:20 |
| clarkb | (I don't expect any issueswith this particular cloud) | 21:21 |
| pabelanger | okay, I'm going to grab some food. Will check back in a few hours and help out | 21:21 |
| *** agopi|out has joined #openstack-infra | 21:22 | |
| *** ldnunes has quit IRC | 21:23 | |
| clarkb | #status log zuul executors have been restarted to pick up latest security fix for localhost execution | 21:23 |
| openstackstatus | clarkb: finished logging | 21:23 |
| *** salv-orlando has quit IRC | 21:24 | |
| *** salv-orlando has joined #openstack-infra | 21:26 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: [WIP] new diskimage-build command with config https://review.openstack.org/554002 | 21:26 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Move fail early checks in to python https://review.openstack.org/555462 | 21:26 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: [WIP] Move environment create/cleanup to python https://review.openstack.org/555463 | 21:26 |
| *** pcichy has quit IRC | 21:27 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: [WIP] Move environment create/cleanup to python https://review.openstack.org/555463 | 21:28 |
| *** agopi|out is now known as agopi | 21:31 | |
| dmsimard | logan-: nodepool VMs should only use the ipv6 network from "Gateway Network" right ? | 21:32 |
| dmsimard | i.e, we take out the flat external network from the clouds.yaml config like so: http://paste.openstack.org/show/709024/ | 21:33 |
| *** aviau has quit IRC | 21:34 | |
| *** aviau has joined #openstack-infra | 21:36 | |
| logan- | Yes they only need gateway net, no flat external for nodepool | 21:37 |
| logan- | They can reach ipv4 over gateway net due to snat | 21:37 |
| dmsimard | ++ | 21:38 |
| *** Goneri has quit IRC | 21:40 | |
| dmsimard | infra-root: I found that nl03.o.o had a peculiar cpu graph: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64813&rra_id=0&view_type=tree&graph_start=1516570740&graph_end=1521754740 | 21:41 |
| dmsimard | It seems like the puppet agent is looping | 21:42 |
| dmsimard | strace is spamming the following http://paste.openstack.org/show/709029/ | 21:42 |
| dmsimard | according to https://tickets.puppetlabs.com/browse/PA-1743 it seems like a ruby issue. | 21:43 |
| *** esberglu has quit IRC | 21:44 | |
| dmsimard | soft kill didn't work, kill -9 went through | 21:44 |
| *** esberglu has joined #openstack-infra | 21:44 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Move fail early checks in to python https://review.openstack.org/555462 | 21:45 |
| dmsimard | It seems like puppet had been running periodically as usual according to syslog | 21:45 |
| *** samuelbeau has joined #openstack-infra | 21:45 | |
| dmsimard | also, it seems like something is wrong with this package, it's being set to "present" on every run: (/Stage[main]/Nodepool/Package[libxslt-dev]/ensure) ensure changed 'purged' to 'present' | 21:45 |
| dmsimard | #status log killed a 21 day old puppet apply on nl03.o.o, was using 100% CPU. strace shown a spam of "sched_yield" and nothing else which seems to have been categorized as a ruby issue in https://tickets.puppetlabs.com/browse/PA-1743 | 21:47 |
| openstackstatus | dmsimard: finished logging | 21:47 |
| *** jamesmcarthur has quit IRC | 21:47 | |
| *** jamesmcarthur has joined #openstack-infra | 21:48 | |
| *** danpawlik has joined #openstack-infra | 21:49 | |
| *** esberglu has quit IRC | 21:49 | |
| *** jamesmcarthur has quit IRC | 21:52 | |
| *** pcaruana has quit IRC | 21:53 | |
| *** danpawlik has quit IRC | 21:54 | |
| *** yamamoto has joined #openstack-infra | 21:55 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone credentials to nodepool's cloud.yaml https://review.openstack.org/555470 | 21:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add limestone to nodepool builder and launcher configuration https://review.openstack.org/555476 | 22:00 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone credentials to nodepool's cloud.yaml https://review.openstack.org/555470 | 22:01 |
| pabelanger | dmsimard: -1 on system-config patch, left comment | 22:01 |
| dmsimard | I saw, thanks | 22:01 |
| dmsimard | pabelanger: replied with a question | 22:03 |
| *** garyk has quit IRC | 22:04 | |
| *** rlandy is now known as rlandy|bbl | 22:04 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone credentials to nodepool's cloud.yaml https://review.openstack.org/555470 | 22:06 |
| pabelanger | replied | 22:06 |
| *** agopi is now known as agopi|out | 22:12 | |
| *** boden has quit IRC | 22:14 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Cleanup references to infracloud, hpcloud and tripleo in site.pp https://review.openstack.org/555478 | 22:15 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add limestone credentials to nodepool's cloud.yaml https://review.openstack.org/555470 | 22:16 |
| *** agopi|out has quit IRC | 22:17 | |
| *** wolverineav has quit IRC | 22:17 | |
| *** wolverineav has joined #openstack-infra | 22:18 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Cleanup references to infracloud, hpcloud and tripleo in site.pp https://review.openstack.org/555478 | 22:19 |
| dmsimard | stepping away for now, stack of patches to look at: https://review.openstack.org/#/q/topic:add-limestone | 22:19 |
| *** danpawlik has joined #openstack-infra | 22:24 | |
| ianw | thanks, looks like a good cleanup. i think i removed a few bits of infracloud | 22:27 |
| *** danpawlik has quit IRC | 22:29 | |
| *** rcernin has joined #openstack-infra | 22:34 | |
| *** dayou has quit IRC | 22:42 | |
| *** dayou has joined #openstack-infra | 22:43 | |
| *** edmondsw has joined #openstack-infra | 22:46 | |
| *** andreas_s has joined #openstack-infra | 22:47 | |
| *** dayou has quit IRC | 22:52 | |
| *** andreas_s has quit IRC | 22:52 | |
| clarkb | I noted some additional cleanup that shoudl probably happen as well. But yay to getting limestone up | 22:54 |
| *** jlabarre has quit IRC | 22:54 | |
| clarkb | dmsimard: also ianw and I have different concerns on the launch node change. I think my suggestion might half address ianw's concern too | 22:54 |
| *** dayou has joined #openstack-infra | 22:55 | |
| ianw | yeah, don't want to bikeshed too much over it, but would probably be good to at least update the instructions | 22:56 |
| *** masber has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add limestone-specific network configuration to *clouds.yaml https://review.openstack.org/555423 | 22:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Test growroot in boot tests https://review.openstack.org/555103 | 23:00 |
| aspiers | where's the code which automatically closes github pull requests on OpenStack repos? | 23:00 |
| *** hongbin has quit IRC | 23:01 | |
| pabelanger | jeepyb | 23:01 |
| ianw | clarkb: ^^ i suspect that might work for testing growroot. | 23:01 |
| pabelanger | aspiers: http://git.openstack.org/cgit/openstack-infra/jeepyb | 23:02 |
| pabelanger | close_pull_requests.py | 23:02 |
| aspiers | pabelanger: thanks! what triggers that? | 23:02 |
| clarkb | ianw: cool, taking a look | 23:02 |
| clarkb | aspiers: cron | 23:02 |
| pabelanger | aspiers: crontab, from system-config | 23:02 |
| aspiers | gotcha | 23:02 |
| *** felipemonteiro has quit IRC | 23:03 | |
| ianw | clarkb: the last run failed because it looked for exactly a 5gb partition, not considering there's the bootloader taking up at least a mb. but the partition did grow | 23:03 |
| pabelanger | aspiers: http://git.openstack.org/cgit/openstack-infra/puppet-github/tree/manifests/init.pp#n88 | 23:03 |
| *** danpawlik has joined #openstack-infra | 23:03 | |
| ianw | pabelanger: https://review.openstack.org/#/c/553894/ what do you think of offsetting these jobs like this? | 23:03 |
| clarkb | ianw: what is the first number in nova flavor-create? the latter 3 are ram, disk, cpu? | 23:03 |
| pabelanger | ianw: is that causing issues? | 23:04 |
| ianw | also a trivial cleanup one https://review.openstack.org/#/c/539828/ to help get your stackalytics stats up :) | 23:04 |
| ianw | pabelanger: everything seems stable at the moment. it still seems like a better idea not to thundering herd on the top of the hour | 23:04 |
| ianw | clarkb: yeah, i had to look it up ... it's name id ram disk vcpus | 23:05 |
| pabelanger | ianw: +2 with comment | 23:05 |
| clarkb | ianw: ah | 23:05 |
| ianw | i think we should change that to openstackclient | 23:05 |
| *** eernst has quit IRC | 23:05 | |
| *** camunoz has quit IRC | 23:06 | |
| aspiers | pabelanger, clarkb: https://github.com/isaacs/github/issues/1191 | 23:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Publish docs to zuul-ci.org https://review.openstack.org/555497 | 23:06 |
| clarkb | ianw: ya probably | 23:06 |
| clarkb | ianw: osc seems to have nice --$arg flags | 23:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add review-dev01.o.o to cacti.o.o https://review.openstack.org/555498 | 23:07 |
| pabelanger | clarkb: ianw: ^thanks for reminder | 23:07 |
| clarkb | aspiers: we aren't the only ones that want to do that? | 23:07 |
| *** danpawlik has quit IRC | 23:07 | |
| aspiers | clarkb: haha no way :) | 23:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Publish docs to zuul-ci.org https://review.openstack.org/555501 | 23:09 |
| pabelanger | :) | 23:09 |
| clarkb | aspiers: also the reason you can't just disable pull requests and keep issues is PRs are issues aiui | 23:10 |
| clarkb | aspiers: in fact if you read the script we use it jumps through some hoops to close the PRs beacuse the api model is weird | 23:10 |
| fungi | aspiers: any support for disabling pull requests would be good. we actually disable issues already but that doesn't disable pull requests | 23:10 |
| clarkb | aspiers: you basically have to find the PR number then effectively cast it as an issue when you talk to the api to clsoe it | 23:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Publish docs to zuul-ci.org https://review.openstack.org/555501 | 23:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Offset cron jobs slightly https://review.openstack.org/553894 | 23:11 |
| aspiers | yeah, their backend seems to mix PRs with issues to some extent | 23:11 |
| *** eernst has joined #openstack-infra | 23:12 | |
| ianw | clarkb: while i'm clearing out ... any further notes on the infra-root email for https://review.openstack.org/#/c/541960/1 ? i think that's about the only mention of it for admins | 23:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Publish zuul-jobs docs to zuul-ci.org https://review.openstack.org/555503 | 23:12 |
| aspiers | I met a GitHubber at the London Git Meetup and complained loudly (but nicely) to him that GitHub engineering needs to stop acting like a black box. Unfortunately he's not in the right role to be able to fix that | 23:12 |
| *** r-daneel has quit IRC | 23:12 | |
| *** masuberu has joined #openstack-infra | 23:13 | |
| clarkb | ianw: fungi is one who is properly set up on that (I continue to fail to get around to imap'ing that account) I'd check if fungi wants any other info in there? | 23:13 |
| ianw | clarkb: ok ... btw i checked logstash-worker16.openstack.org after the incident notification they sent to there about it, it seems ok | 23:14 |
| fungi | i believe ianw is also reading it now | 23:14 |
| fungi | as evidenced by ^ | 23:15 |
| ianw | fungi: there isn't a "rax" folder under clouds? should we add one and file the incident reports there? | 23:16 |
| fungi | no objection from me | 23:16 |
| ianw | ok, that way we can file them when someone's at least had a look | 23:17 |
| fungi | sounds good | 23:17 |
| *** masber has quit IRC | 23:17 | |
| *** masber has joined #openstack-infra | 23:18 | |
| *** masuberu has quit IRC | 23:18 | |
| ianw | clarkb: do you have a playbook already for restarting logstash etc, https://review.openstack.org/#/c/512467/1/doc/source/logstash.rst ? | 23:19 |
| clarkb | ianw: no because half the time jvm refuses to restart :/ | 23:20 |
| clarkb | so I end up wrnagling it by hand anyways. That said I think a config change largely addressed that problem | 23:20 |
| ianw | pabelanger / clarkb: another cleanup for nodepool puppet stuff; just make it zk https://review.openstack.org/#/c/547248/ | 23:20 |
| ianw | clarkb: ok, do you think that note is useful, or will i just abandon? | 23:20 |
| clarkb | ianw: I've not seen a need for restarting things since we upadted the config to timeout connections | 23:21 |
| pabelanger | yah, I think I'm going to try moving zookeeper data into zk cluster tomorrow | 23:21 |
| clarkb | ianw: the problem aiui was elasticsaerch server would go away. Logsatsh would be blocking on pushing content to that particular ES server and never stop | 23:21 |
| pabelanger | then manually test the data is good | 23:21 |
| clarkb | ianw: but since the config update to timeout es connections I think its been pretty stable | 23:21 |
| pabelanger | we can then schedule migration of services | 23:22 |
| *** e0ne has joined #openstack-infra | 23:22 | |
| ianw | corvus: you're probably the other person who has reprepro'd manually ... do you think this no-timeout flag will be helpful -> https://review.openstack.org/#/c/550683 | 23:24 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove mirror01.dfw.o.o https://review.openstack.org/539828 | 23:26 |
| ianw | clarkb / pabelanger : i think making the gerrit jobs in-tree should be fairly straight forward and worth doing before an upgrade? https://review.openstack.org/#/c/552718/ | 23:26 |
| *** Swami has quit IRC | 23:26 | |
| clarkb | ianw: ya I left a question on that change about copying artifacts in untrusted jobs | 23:27 |
| clarkb | (because running in check) | 23:27 |
| ianw | oh right, sorry, missed that | 23:27 |
| corvus | ianw: reprepro +3 | 23:28 |
| clarkb | I want to say that came up in a different context and the answer was basiaclly use the stock logs copying | 23:28 |
| clarkb | (since that is defined in a trusted context we can consume logs publishing from an untrusted context but we can't define our own synchronize in an untrusted context?) | 23:28 |
| *** masber has quit IRC | 23:29 | |
| clarkb | of course now that I look at it more closely this apepars to be using the stock copy artifacts things for me and doesn't touch the fielserver copy so maybe its fine | 23:29 |
| *** masber has joined #openstack-infra | 23:29 | |
| ianw | well initially it's just moving the legacy playbooks in tree | 23:29 |
| clarkb | ya but the legacy jobs were in trusted context whcih doubles my confusion around this :) | 23:30 |
| *** eernst has quit IRC | 23:32 | |
| ianw | hmm, we changed gerrit-buck to use legacy-base, and it's not copying artifacts anywhere i think now, right ? https://review.openstack.org/#/c/552698/1/zuul.d/zuul-legacy-jobs.yaml | 23:32 |
| ianw | we discussed that we should copy the .war into the logs so it could be played with ... previously the check job used to automatically push that up to tarballs? | 23:33 |
| *** danpawlik has joined #openstack-infra | 23:33 | |
| ianw | "gerrit-package-buck" the gate job ... probably hasn't run yet? not sure anything has got that far | 23:34 |
| clarkb | ianw: I guess we can throw this at zuul and see if it breaks | 23:35 |
| *** kjackal has quit IRC | 23:36 | |
| *** kjackal has joined #openstack-infra | 23:36 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add a note on the shared infra root mail account https://review.openstack.org/541960 | 23:36 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix codesearch stderr capturing https://review.openstack.org/529188 | 23:36 |
| *** danpawlik has quit IRC | 23:38 | |
| clarkb | ianw: the rsync opts there are another thing that throw me off. We should maybe try to specify the file we actually want in the future | 23:39 |
| clarkb | ianw: in any case +2 | 23:40 |
| ianw | pabelanger: do you want to merge https://review.openstack.org/#/c/547248/ as a small cleanup before you do zk cluster? | 23:40 |
| *** caphrim007 has quit IRC | 23:41 | |
| *** jtomasek has quit IRC | 23:42 | |
| pabelanger | wfm | 23:42 |
| clarkb | ok back to packing now | 23:42 |
| *** tosky has quit IRC | 23:43 | |
| *** eernst has joined #openstack-infra | 23:44 | |
| *** calebb has quit IRC | 23:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add a documentation index page https://review.openstack.org/555505 | 23:51 |
| *** calebb has joined #openstack-infra | 23:52 | |
| *** e0ne_ has joined #openstack-infra | 23:54 | |
| *** danpawlik has joined #openstack-infra | 23:54 | |
| ianw | hmm, going back to things on my todo list and checking graphite logging ... it seems borked | 23:54 |
| *** esberglu has joined #openstack-infra | 23:54 | |
| ianw | carbon-ca 1398 www-data 6u REG 202,1 4197083575 917759 /var/log/graphite/carbon-cache-a/console.log.1 (deleted) | 23:55 |
| ianw | i guess it's not getting the hint to start a new file | 23:55 |
| *** e0ne has quit IRC | 23:57 | |
| *** xinliang has quit IRC | 23:58 | |
| *** esberglu has quit IRC | 23:59 | |
| ianw | i bet that i never restarted it after https://review.openstack.org/#/c/541488/ doesn't help. i'll try that | 23:59 |
| *** danpawlik has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!