| *** tosky has quit IRC | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:12 | |
| *** yamamoto has quit IRC | 00:17 | |
| *** imacdonn has quit IRC | 00:58 | |
| *** dave-mccowan has joined #openstack-infra | 01:03 | |
| *** yamamoto has joined #openstack-infra | 01:06 | |
| *** yamamoto has quit IRC | 01:13 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** dave-mccowan has quit IRC | 01:18 | |
| *** dulek has quit IRC | 01:23 | |
| *** dulek has joined #openstack-infra | 01:23 | |
| *** agopi_ has quit IRC | 01:31 | |
| *** agopi_ has joined #openstack-infra | 01:32 | |
| *** HeOS has quit IRC | 01:36 | |
| *** r-daneel has joined #openstack-infra | 01:36 | |
| *** agopi_ has quit IRC | 01:41 | |
| *** agopi_ has joined #openstack-infra | 01:41 | |
| *** salv-orlando has joined #openstack-infra | 01:46 | |
| *** salv-orl_ has quit IRC | 01:49 | |
| *** dulek has quit IRC | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:50 | |
| *** dmellado has quit IRC | 01:51 | |
| *** stevebaker has quit IRC | 01:51 | |
| *** dmellado has joined #openstack-infra | 01:51 | |
| *** zoli has quit IRC | 01:51 | |
| *** gouthamr has quit IRC | 01:51 | |
| *** dulek has joined #openstack-infra | 01:54 | |
| *** stevebaker has joined #openstack-infra | 01:55 | |
| *** gouthamr has joined #openstack-infra | 01:58 | |
| *** zoli has joined #openstack-infra | 01:59 | |
| *** yamamoto has joined #openstack-infra | 02:09 | |
| *** yamamoto has quit IRC | 02:15 | |
| *** eernst has joined #openstack-infra | 02:57 | |
| *** eernst has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-infra | 03:06 | |
| *** yamamoto has joined #openstack-infra | 03:11 | |
| *** yamamoto has quit IRC | 03:17 | |
| *** ihar has joined #openstack-infra | 03:17 | |
| *** aviau has quit IRC | 03:18 | |
| *** aviau has joined #openstack-infra | 03:18 | |
| *** ihar has quit IRC | 03:32 | |
| *** ihar has joined #openstack-infra | 03:37 | |
| *** janki has joined #openstack-infra | 03:39 | |
| *** sambetts_ has quit IRC | 03:42 | |
| *** sambetts_ has joined #openstack-infra | 03:43 | |
| *** sambetts_ has quit IRC | 03:54 | |
| *** sambetts_ has joined #openstack-infra | 03:56 | |
| *** mriedem has quit IRC | 03:59 | |
| *** pbourke_ has quit IRC | 04:04 | |
| *** pbourke_ has joined #openstack-infra | 04:04 | |
| *** hongbin has quit IRC | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:07 | |
| *** jchhatbar has joined #openstack-infra | 04:10 | |
| *** yamamoto has joined #openstack-infra | 04:13 | |
| *** janki has quit IRC | 04:13 | |
| *** dhajare has joined #openstack-infra | 04:17 | |
| *** jchhatba_ has joined #openstack-infra | 04:17 | |
| *** hwoarang has quit IRC | 04:18 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** jchhatbar has quit IRC | 04:20 | |
| *** jamesmcarthur has joined #openstack-infra | 04:26 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** snapiri has joined #openstack-infra | 04:34 | |
| *** jogo has quit IRC | 05:01 | |
| *** dsariel has joined #openstack-infra | 05:08 | |
| *** jogo has joined #openstack-infra | 05:13 | |
| *** psachin has joined #openstack-infra | 05:13 | |
| *** yamamoto has joined #openstack-infra | 05:15 | |
| *** yamamoto has quit IRC | 05:19 | |
| *** ihar has quit IRC | 05:21 | |
| dirk | pabelanger: yw | 05:48 |
|---|---|---|
| *** oanson has joined #openstack-infra | 05:52 | |
| *** armaan has joined #openstack-infra | 05:53 | |
| *** e0ne has joined #openstack-infra | 05:54 | |
| *** sambetts_ has quit IRC | 05:54 | |
| *** sambetts_ has joined #openstack-infra | 05:56 | |
| *** oanson has quit IRC | 06:01 | |
| *** oanson has joined #openstack-infra | 06:02 | |
| *** e0ne has quit IRC | 06:03 | |
| *** pcichy has joined #openstack-infra | 06:14 | |
| *** yamamoto has joined #openstack-infra | 06:15 | |
| *** iranzo has joined #openstack-infra | 06:17 | |
| *** yamamoto has quit IRC | 06:20 | |
| *** dhajare has quit IRC | 06:26 | |
| *** dbecker has quit IRC | 06:32 | |
| *** lpetrut has joined #openstack-infra | 06:38 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** dhajare has joined #openstack-infra | 06:59 | |
| *** yamamoto has joined #openstack-infra | 07:00 | |
| *** masber has joined #openstack-infra | 07:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: unpause Gentoo https://review.openstack.org/561285 | 07:10 |
| *** armaan has quit IRC | 07:26 | |
| *** slaweq has joined #openstack-infra | 07:27 | |
| *** jchhatba_ has quit IRC | 07:31 | |
| *** janki has joined #openstack-infra | 07:31 | |
| *** dpawlik has quit IRC | 07:33 | |
| *** sambetts_ has quit IRC | 07:36 | |
| *** sambetts_ has joined #openstack-infra | 07:38 | |
| *** janki has quit IRC | 07:40 | |
| *** armaan has joined #openstack-infra | 07:48 | |
| *** lpetrut has quit IRC | 07:58 | |
| *** agopi_ has quit IRC | 08:09 | |
| *** lihi has joined #openstack-infra | 08:16 | |
| *** salv-orl_ has joined #openstack-infra | 08:18 | |
| *** salv-orlando has quit IRC | 08:18 | |
| *** lpetrut has joined #openstack-infra | 08:24 | |
| *** nmagnezi has left #openstack-infra | 08:31 | |
| *** lpetrut has quit IRC | 08:34 | |
| *** lpetrut has joined #openstack-infra | 08:34 | |
| *** yamamoto has quit IRC | 08:37 | |
| *** lpetrut has quit IRC | 08:39 | |
| *** pas-ha has joined #openstack-infra | 08:40 | |
| *** dbecker has quit IRC | 08:55 | |
| *** lpetrut has joined #openstack-infra | 08:56 | |
| *** psachin has quit IRC | 09:05 | |
| *** armaan has quit IRC | 09:17 | |
| *** jamesmcarthur has joined #openstack-infra | 09:27 | |
| *** jamesmcarthur has quit IRC | 09:31 | |
| *** yamamoto has joined #openstack-infra | 09:38 | |
| *** armaan has joined #openstack-infra | 09:38 | |
| *** pcaruana has joined #openstack-infra | 09:38 | |
| *** slaweq has quit IRC | 09:39 | |
| *** slaweq has joined #openstack-infra | 09:39 | |
| *** tom-shan has joined #openstack-infra | 09:41 | |
| *** lpetrut has quit IRC | 09:41 | |
| *** yamamoto has quit IRC | 09:44 | |
| *** slaweq has quit IRC | 09:44 | |
| *** tom-shan has quit IRC | 09:44 | |
| *** yamamoto has joined #openstack-infra | 10:16 | |
| *** lpetrut has joined #openstack-infra | 10:34 | |
| *** armaan has quit IRC | 10:38 | |
| *** HeOS has joined #openstack-infra | 10:45 | |
| *** xinliang has quit IRC | 10:48 | |
| *** hwoarang has joined #openstack-infra | 10:57 | |
| *** xinliang has joined #openstack-infra | 11:00 | |
| *** jrist has quit IRC | 11:27 | |
| *** jrist has joined #openstack-infra | 11:27 | |
| *** yamamoto has quit IRC | 11:28 | |
| *** hwoarang has quit IRC | 11:28 | |
| *** hwoarang has joined #openstack-infra | 11:28 | |
| *** krenczewski has quit IRC | 11:36 | |
| *** slaweq has joined #openstack-infra | 11:54 | |
| *** slaweq has quit IRC | 11:59 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config master: Remove legacy job: legacy-install-dsvm-dragonflow-kuryr-kubernetes https://review.openstack.org/561459 | 12:27 |
| openstackgerrit | Omer Anson proposed openstack-infra/openstack-zuul-jobs master: Remove legacy job: install-dsvm-dragonflow-kuryr-kubernetes https://review.openstack.org/561460 | 12:29 |
| *** yamamoto has quit IRC | 12:32 | |
| mordred | pabelanger: projects using stestr install it as part of test-requirements.txt - however, the thing we get in os-testr fromthat venv is subunit2html that we use for reports | 12:34 |
| mordred | pabelanger: roles/fetch-subunit-output/tasks/find-subunit-html.yaml in zuul-jobs | 12:34 |
| mordred | pabelanger: if we add something to that role to install subunit2html if it's not found... like an ensure-subunit2html role - then I think we can get rid of that venv | 12:35 |
| *** sshnaidm|off has quit IRC | 12:38 | |
| *** HeOS has quit IRC | 12:41 | |
| *** dpawlik has joined #openstack-infra | 12:46 | |
| *** dpawlik has quit IRC | 12:51 | |
| *** sambetts_ has quit IRC | 12:55 | |
| *** sambetts_ has joined #openstack-infra | 12:55 | |
| *** lpetrut has quit IRC | 12:55 | |
| *** dayou has quit IRC | 13:17 | |
| *** dayou has joined #openstack-infra | 13:18 | |
| *** lpetrut has joined #openstack-infra | 13:26 | |
| *** jamesmcarthur has joined #openstack-infra | 13:28 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| *** sambetts_ has quit IRC | 13:36 | |
| *** gema has quit IRC | 13:38 | |
| *** sambetts_ has joined #openstack-infra | 13:39 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** gema has joined #openstack-infra | 13:40 | |
| *** gema has joined #openstack-infra | 13:40 | |
| *** d0ugal has quit IRC | 13:46 | |
| *** sshnaidm|off has joined #openstack-infra | 13:55 | |
| *** d0ugal has joined #openstack-infra | 13:56 | |
| *** nicolasbock has joined #openstack-infra | 14:10 | |
| *** sambetts_ has quit IRC | 14:25 | |
| *** sambetts_ has joined #openstack-infra | 14:27 | |
| *** sshnaidm|off is now known as sshnaidm | 14:31 | |
| *** slaweq has joined #openstack-infra | 14:35 | |
| *** yamamoto has joined #openstack-infra | 14:35 | |
| *** slaweq has quit IRC | 14:39 | |
| *** yamamoto has quit IRC | 14:41 | |
| *** bobh has joined #openstack-infra | 14:45 | |
| *** lpetrut has quit IRC | 14:51 | |
| *** r-daneel has quit IRC | 14:56 | |
| *** sambetts_ has quit IRC | 14:59 | |
| *** sambetts_ has joined #openstack-infra | 15:00 | |
| *** lpetrut has joined #openstack-infra | 15:03 | |
| *** jbadiapa has joined #openstack-infra | 15:04 | |
| *** d0ugal has quit IRC | 15:13 | |
| *** VW has joined #openstack-infra | 15:20 | |
| *** jtomasek has joined #openstack-infra | 15:21 | |
| *** dsariel has quit IRC | 15:31 | |
| pabelanger | mordred: ah, right thanks! I'll maybe look into that this week | 15:31 |
| *** yamamoto has joined #openstack-infra | 15:37 | |
| *** snapiri has quit IRC | 15:38 | |
| *** dhajare has quit IRC | 15:39 | |
| *** d0ugal has joined #openstack-infra | 15:40 | |
| *** yamamoto has quit IRC | 15:42 | |
| *** snapiri has joined #openstack-infra | 15:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "unpause Gentoo" https://review.openstack.org/561466 | 15:44 |
| pabelanger | AJaeger: prometheanfire: ^ gentoo hasn't been setup to work with project-config elements, so we'll need to pause again | 15:48 |
| *** psachin has joined #openstack-infra | 15:51 | |
| pabelanger | prometheanfire: ianw: also, it seems pip-and-virtualenv for gentoo doesn't actually install get-pip.py for source installs. I think we need to fix that to bring gentoo images online, as we want to have the same version of pip across all our images | 15:52 |
| pabelanger | eg: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n106 | 15:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "unpause Gentoo" https://review.openstack.org/561466 | 15:59 |
| *** iranzo has quit IRC | 16:00 | |
| *** lpetrut has quit IRC | 16:08 | |
| *** lpetrut has joined #openstack-infra | 16:09 | |
| *** VW has quit IRC | 16:09 | |
| *** VW has joined #openstack-infra | 16:19 | |
| *** VW has quit IRC | 16:29 | |
| *** slaweq has joined #openstack-infra | 16:33 | |
| *** yamamoto has joined #openstack-infra | 16:38 | |
| *** slaweq has quit IRC | 16:43 | |
| *** yamamoto has quit IRC | 16:43 | |
| *** lbragstad has joined #openstack-infra | 16:52 | |
| *** dsariel has joined #openstack-infra | 16:53 | |
| *** dhajare has joined #openstack-infra | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| *** jbadiapa has quit IRC | 17:07 | |
| *** jbadiapa has joined #openstack-infra | 17:08 | |
| snapiri | Hi Infra, cross-posting here from dev: a new issue is failing builds. It seems like the pip behaviour (at least on ubuntu) has changed, and now it is failing on uninstall of distutils packages | 17:12 |
| snapiri | see: http://logs.openstack.org/41/554241/24/gate/dragonflow-dsvm-fullstack-redis/ef5da31/logs/devstacklog.txt.gz#_2018-04-15_09_09_03_094 | 17:12 |
| snapiri | it succeeded until today: http://logs.openstack.org/41/554241/24/check/dragonflow-dsvm-fullstack-redis/f988d50/logs/devstacklog.txt.gz#_2018-04-11_11_28_53_075 | 17:12 |
| snapiri | from what I see the only problematic package is psutil, and I suspect it is because of https://github.com/openstack-dev/devstack/blob/master/files/debs/dstat#L2 | 17:12 |
| snapiri | could anyone please confirm? | 17:12 |
| clarkb | snapiri: yes its due to the pip 10 release. We've known about it for a while now and there are a couple dev mailing list thread about it | 17:14 |
| snapiri | this is the price of backlogging on the mailing list :) | 17:15 |
| snapiri | for me | 17:15 |
| snapiri | thanks! | 17:15 |
| *** bobh has quit IRC | 17:19 | |
| *** dsariel has quit IRC | 17:29 | |
| *** lbragstad has quit IRC | 17:30 | |
| *** bobh has joined #openstack-infra | 17:31 | |
| *** jamesmcarthur has quit IRC | 17:33 | |
| *** bobh has quit IRC | 17:36 | |
| *** sshnaidm has quit IRC | 17:38 | |
| *** yamamoto has joined #openstack-infra | 17:40 | |
| *** snapiri has quit IRC | 17:43 | |
| *** yamamoto has quit IRC | 17:46 | |
| fungi | snapiri: also, it's mainly an issue for devstack-based jobs, so a more appropriate irc channel for those discussions is probably #openstack-qa | 17:46 |
| *** bobh has joined #openstack-infra | 17:46 | |
| *** bobh has quit IRC | 17:50 | |
| *** sshnaidm has joined #openstack-infra | 17:54 | |
| *** d0ugal has quit IRC | 17:55 | |
| *** psachin has quit IRC | 17:55 | |
| *** bobh has joined #openstack-infra | 17:56 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| *** bobh_ has joined #openstack-infra | 18:00 | |
| *** bobh has quit IRC | 18:00 | |
| *** pcichy has quit IRC | 18:02 | |
| *** e0ne has joined #openstack-infra | 18:05 | |
| *** bobh_ has quit IRC | 18:05 | |
| *** d0ugal has joined #openstack-infra | 18:09 | |
| *** bobh has joined #openstack-infra | 18:12 | |
| *** bobh has quit IRC | 18:16 | |
| *** hongbin has joined #openstack-infra | 18:20 | |
| *** bobh has joined #openstack-infra | 18:21 | |
| *** bobh has quit IRC | 18:26 | |
| prometheanfire | pabelanger: ok, are there bugs? | 18:28 |
| *** bobh has joined #openstack-infra | 18:34 | |
| *** slagle has joined #openstack-infra | 18:37 | |
| *** bobh has quit IRC | 18:39 | |
| *** yamamoto has joined #openstack-infra | 18:42 | |
| *** bobh has joined #openstack-infra | 18:44 | |
| *** yamamoto has quit IRC | 18:47 | |
| *** bobh has quit IRC | 18:49 | |
| *** bobh has joined #openstack-infra | 18:52 | |
| *** jamesmcarthur has quit IRC | 18:53 | |
| *** jamesmcarthur has joined #openstack-infra | 18:54 | |
| pabelanger | prometheanfire: yah, should be able to reproduce with tools/build-image.sh in project-config | 18:54 |
| pabelanger | 2018-04-15 15:37:10.971 | Failed to enable unit, unit iptables.service does not exist. | 18:54 |
| pabelanger | was error | 18:54 |
| pabelanger | but, http://nb01.openstack.org should have logs | 18:54 |
| pabelanger | but seems down for me | 18:54 |
| pabelanger | clarkb: fungi: are you able to hit HTTP on nb01.o.o or nb02.o.o ? | 18:55 |
| prometheanfire | pabelanger: ok, I wonder if iptables wasn't installed or something | 18:56 |
| prometheanfire | /lib/systemd/system/iptables-restore.service | 18:56 |
| prometheanfire | I guess on gentoo we use a diferent name (store/restore variants of ip and ip6) | 18:56 |
| pabelanger | prometheanfire: let me get a build log, since http isn't working | 18:56 |
| *** bobh has quit IRC | 18:57 | |
| pabelanger | prometheanfire: https://pabelanger.fedorapeople.org/gentoo-17-0-systemd-0000000757.log | 19:00 |
| pabelanger | prometheanfire: let me know when you have it | 19:00 |
| prometheanfire | thanks | 19:00 |
| prometheanfire | saved | 19:00 |
| pabelanger | kk, removing | 19:00 |
| prometheanfire | so, that's a dib thing (the iptables)? | 19:01 |
| pabelanger | yah, it is our elements in project-config: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/ | 19:01 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh will be an easy way to build it locally and test | 19:02 |
| prometheanfire | ok, those then | 19:02 |
| prometheanfire | pabelanger: so I imagine you want ip6tables too? | 19:02 |
| *** bobh has joined #openstack-infra | 19:04 | |
| pabelanger | prometheanfire: yah, we setup both ipv4 / ipv6 rules on images | 19:04 |
| prometheanfire | ok | 19:04 |
| prometheanfire | so I have 4 services to enable | 19:04 |
| *** iranzo has joined #openstack-infra | 19:07 | |
| *** iranzo has quit IRC | 19:07 | |
| *** iranzo has joined #openstack-infra | 19:07 | |
| *** jtomasek has quit IRC | 19:07 | |
| *** bobh has quit IRC | 19:08 | |
| *** dhajare has quit IRC | 19:09 | |
| prometheanfire | this should do it | 19:10 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: enable the right iptables services on gentoo https://review.openstack.org/561472 | 19:10 |
| *** slaweq has joined #openstack-infra | 19:11 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: enable the right iptables services on gentoo https://review.openstack.org/561472 | 19:12 |
| openstackgerrit | Merged openstack-infra/system-config master: Add statusbot and enable logs for oooq IRC channel https://review.openstack.org/559999 | 19:15 |
| pabelanger | prometheanfire: cool, do you mind building the image to with patch to ensure everything else works properlly | 19:17 |
| pabelanger | properly* | 19:17 |
| prometheanfire | ya, I'm working on it now (build box needs some slight updating | 19:17 |
| pabelanger | great | 19:18 |
| *** bobh has joined #openstack-infra | 19:18 | |
| prometheanfire | it looks like I still need to define the profile export for gentoo outside that script though | 19:20 |
| *** claudiub|2 has quit IRC | 19:21 | |
| *** HeOS has joined #openstack-infra | 19:22 | |
| *** bobh has quit IRC | 19:23 | |
| prometheanfire | looks like the script runs cache-devstack and the gentoo image by default does not | 19:23 |
| prometheanfire | probably because it's not supposed to run devstack at this point, we'll see if it hurts anything I guess | 19:24 |
| prometheanfire | anyway, building now | 19:24 |
| *** pcaruana has quit IRC | 19:24 | |
| *** armaan has joined #openstack-infra | 19:28 | |
| *** bobh has joined #openstack-infra | 19:33 | |
| *** imacdonn has joined #openstack-infra | 19:35 | |
| *** bobh has quit IRC | 19:38 | |
| *** slagle has quit IRC | 19:38 | |
| *** openstackstatus has quit IRC | 19:41 | |
| *** openstack has joined #openstack-infra | 19:43 | |
| *** ChanServ sets mode: +o openstack | 19:43 | |
| *** yamamoto has joined #openstack-infra | 19:43 | |
| *** bobh has quit IRC | 19:47 | |
| *** yamamoto has quit IRC | 19:48 | |
| *** bobh has joined #openstack-infra | 19:52 | |
| *** dpawlik has joined #openstack-infra | 19:56 | |
| *** bobh has quit IRC | 19:57 | |
| *** dave-mccowan has joined #openstack-infra | 20:02 | |
| pabelanger | ianw: clarkb: fungi: yah, Centos DIBs are failing because of pip10 | 20:03 |
| pabelanger | 2018-04-15 18:09:56.730 | Cannot uninstall 'virtualenv'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. | 20:03 |
| pabelanger | have no idea how to best solve that | 20:03 |
| pabelanger | unless we pip install --user virtualenv | 20:04 |
| pabelanger | then update everything to use that | 20:04 |
| *** claudiub|2 has joined #openstack-infra | 20:05 | |
| *** nicolasbock has quit IRC | 20:05 | |
| *** e0ne has quit IRC | 20:06 | |
| fungi | pabelanger: any reason 1. that we're preinstalling virtualenv from rpm and 2. that we're attempting to pip upgrade it? | 20:07 |
| fungi | seems like we should only be doing one or the other in our image builds | 20:08 |
| fungi | not installing the virtualenv rpm is probably the simplest solution if we can figure out what's causing it to get installed in the first place | 20:09 |
| pabelanger | fungi: I think we preinstall them to satisfy yum / dnf that virtualenv installed, then we overwrite it with a newer version. If we don't, I think the issue becomes, when somebody yum installs a python package that needs python-vritualenv, it will then overwrite our version installed via get-pip.py. | 20:11 |
| clarkb | ya that | 20:12 |
| clarkb | but hard to.avoid now | 20:12 |
| pabelanger | right | 20:12 |
| *** yamamoto has joined #openstack-infra | 20:12 | |
| *** d0ugal_ has joined #openstack-infra | 20:12 | |
| fungi | why would an rpm of a python library need virtualenv? | 20:13 |
| fungi | they just install into the system context anyway, right? | 20:13 |
| pabelanger | I am unsure, ianw likely best to answer that | 20:14 |
| pabelanger | we do the same for APT too | 20:14 |
| fungi | seems like a manufactured problem to me, but i'm probably just missing some nuance there | 20:14 |
| *** d0ugal has quit IRC | 20:14 | |
| clarkb | on centos there is no split like with debuntu | 20:15 |
| fungi | anyway, if we have stale images for a week or two while we work through pip 10 related image build failures, that's probably not the end of the world. nodepool protects us nicely in such a situation | 20:15 |
| fungi | clarkb: explicit split of what? virtualenv is a completely separate python package | 20:15 |
| clarkb | /usr/bin/virtualenv and /usr/local/bin/virtualenv | 20:16 |
| clarkb | on ubuntu its a total nonnissue I think | 20:16 |
| fungi | well, if the ubuntu deb of virtualenv is built using distutils then it would be an issue all the same | 20:16 |
| clarkb | with pip10 yes I mean the reason for preinstalling system packages | 20:17 |
| clarkb | on ubuntu just stop preinstalling and its fine. On centos you have have trouble if you install system virtualenv later | 20:17 |
| pabelanger | https://review.openstack.org/327472 | 20:18 |
| *** bobh has joined #openstack-infra | 20:18 | |
| pabelanger | that explains why we do it | 20:18 |
| clarkb | that all said I think with pip10 we cant mix them at all | 20:18 |
| clarkb | so if you do prepare for fallout | 20:19 |
| pabelanger | so, we if update devstack to stop using package version of pip / virtualenv, etc, I think we might be able to revert it. But still doesn't protect of a job installing them and overwriting it | 20:19 |
| fungi | oh, right i guess maybe we should uninstall whatever virtualenv package at the end of image builds once we're done creating virtual envs for various tools? that way jobs can install whichever one they want with no risk of conflict | 20:19 |
| fungi | or do like i do on my personal systems... use distro-packaged virtualenv to bootstrap a virtualenv in which you pip install virtualenv (and then optionally uninstall the distro package of it once you do that) and use the virtualenv'd virtualenv to create other envs | 20:21 |
| fungi | i also install tox into a virtualenv and call that rather than installing it globally | 20:21 |
| *** bobh has quit IRC | 20:22 | |
| pabelanger | yah, that seems like the way forward now | 20:22 |
| fungi | just put a symlink (from ~/bin/tox or /usr/local/bin/tox) into $venvdir/bin/tox | 20:22 |
| clarkb | devstack doesnt use system virtualenv or pip iirc | 20:23 |
| *** bobh has joined #openstack-infra | 20:23 | |
| pabelanger | I guess we could also black list packages in apt / yum to prevent them from overwriting our version from get-pip.py | 20:24 |
| fungi | i basically just maintain a virtualenv symlink farm and never `sudo pip install ...` anything at all | 20:24 |
| clarkb | however it does assume virtualenv exists if running in infra | 20:24 |
| fungi | with the symlimk trick, we can provide a global virtualenv entrypoint executable without distro package or `sudo pip install ...` | 20:25 |
| clarkb | fungi: the problems we'll have are we have to use site system packages and educate on that. Also so many random things assume weord stuff | 20:25 |
| clarkb | like ironic and virtualbmc needing to su to get group membership updated which breaks PATH | 20:26 |
| fungi | yep, i'm sure it won't be straightforward and there will be ugly jagged edges | 20:26 |
| fungi | does it stop having /usr/local/bin in its path when it does that? | 20:26 |
| *** dpawlik has quit IRC | 20:27 | |
| clarkb | no but you can symlink that to your venv it would have to be a different path to avoid breaking other things that use that path right? | 20:28 |
| clarkb | (thinking stuff like gems or npm) | 20:28 |
| *** lbragstad has joined #openstack-infra | 20:28 | |
| *** bobh has quit IRC | 20:28 | |
| clarkb | the other issue is python2 and python3 cant live in one venv so you need two | 20:28 |
| clarkb | and then some way to select into the right one | 20:29 |
| fungi | sure, though that's not really an issue for, e.g., tox | 20:30 |
| pabelanger | yah, maybe we drop /usr/local/bin/virtualenv and have people juse python2 -m virtualenv / python3 -m virtualenv if they need to create it | 20:31 |
| fungi | even if you used sudo pip install or a distro package, the first entrypoint in your execution path will only ever use one python version | 20:31 |
| pabelanger | pip2 / pip3 are still things | 20:31 |
| fungi | right | 20:31 |
| clarkb | fungi: ya the problem is when you want swift under python2 and evrrything else under python3 | 20:32 |
| fungi | this is not really a pip 10 issue though, it's exactly the same under earlier pip | 20:32 |
| clarkb | you cant just replace /usr/local/bin in that case | 20:32 |
| fungi | it really depends on what python application (not library) you're talking about | 20:33 |
| *** VW has joined #openstack-infra | 20:33 | |
| pabelanger | is python -m pip install a thing? | 20:33 |
| *** iranzo has quit IRC | 20:33 | |
| *** bobh has joined #openstack-infra | 20:33 | |
| pabelanger | looks like it is | 20:34 |
| fungi | yeah | 20:34 |
| clarkb | thr approach I tried to take in devstack was python3env/bin was ahead of python2env/bin in PATH | 20:34 |
| clarkb | but vouldnt replace /usr/local/bin with one or the other | 20:35 |
| *** armaan has quit IRC | 20:35 | |
| clarkb | then ran out of time trying to convinve ironic to run virtualbmc in way that didnt break PATH | 20:35 |
| *** armaan has joined #openstack-infra | 20:35 | |
| *** bobh has quit IRC | 20:38 | |
| fungi | if you `virtualenv -p python2.7 /opt/stack/python2venv && /opt/stack/python2venv/bin/pip install /opt/stack/swift && ln -s /opt/stack/venvs/swift/bin/swift /usr/local/bin/` or whatever you can control it that way, right? | 20:39 |
| ianw | pabelanger: or we just rm selected files to override pip | 20:40 |
| ianw | override pip's check | 20:40 |
| clarkb | fungi: only if you know every single command that needs to be symlinked | 20:40 |
| fungi | fair | 20:41 |
| fungi | ianw: genius! fake pip into thinking it's not installed so it will just shut up and get on with it | 20:43 |
| *** lpetrut has quit IRC | 20:44 | |
| ianw | that's my first instinct here, because that's basically what it used to do anyway | 20:44 |
| pabelanger | how would we do that? | 20:44 |
| ianw | rm /usr/lib/python2.7/site-packages/pip-* ; get-pip.py | 20:45 |
| ianw | approximately | 20:45 |
| ianw | pabelanger: was there any progress on the glean xenial issues, or is that still outstanding too? | 20:46 |
| pabelanger | ianw: yah, that is still outstanding. I was going to look more into it in the morning | 20:46 |
| *** jbadiapa has quit IRC | 20:46 | |
| ianw | that has me seriously stumped right now. all i can think is that it is udev not activating the rule | 20:47 |
| *** jbadiapa has joined #openstack-infra | 20:47 | |
| ianw | but afaict, the version of everything between the bad and good images are the same (udev is all part of systemd packages, and didn't update) | 20:48 |
| ianw | hmm, why is http://nb01.openstack.org/ not showing me logs ... | 20:49 |
| *** d0ugal_ has quit IRC | 20:50 | |
| pabelanger | ianw: yah, it is down for me too | 20:51 |
| pabelanger | noted it in backscroll | 20:51 |
| ianw | ACCEPT tcp -- 172.24.4.0/23 anywhere tcp dpt:http | 20:54 |
| *** d0ugal_ has joined #openstack-infra | 20:56 | |
| *** bobh has joined #openstack-infra | 20:57 | |
| ianw | also the rules on nb03 are different | 20:58 |
| *** bobh has quit IRC | 21:00 | |
| *** bobh has joined #openstack-infra | 21:00 | |
| ianw | https://github.com/open-switch/infra_system-config/blob/master/modules/openstack_project/manifests/single_use_slave.pp#L27 | 21:01 |
| *** armaan has quit IRC | 21:01 | |
| ianw | sets up ports 8000, 8003, 8004 for devstack bits | 21:01 |
| ianw | ACCEPT tcp -- 172.24.4.0/23 anywhere tcp dpt:8000 | 21:02 |
| ianw | ACCEPT tcp -- 172.24.4.0/23 anywhere tcp dpt:8003 | 21:02 |
| ianw | ACCEPT tcp -- 172.24.4.0/23 anywhere tcp dpt:8004 | 21:02 |
| ianw | we've run devstack in the builders???? | 21:02 |
| *** slaweq has quit IRC | 21:05 | |
| *** l4yerffej has quit IRC | 21:05 | |
| *** bobh has quit IRC | 21:05 | |
| clarkb | devstacks image listing for caching may have had unexpected side effects? | 21:09 |
| ianw | well i've restarted netfilter-persistent ... if this comes back however, we have some sort of chroot escape i guess :/ | 21:09 |
| ianw | i can't see any simple smoking gun in auth.log where we've sudo run something | 21:09 |
| ianw | and it hasn't happened on nb03.o.o ... and that only builds xenial and debian, so it's likely something *other* that those two | 21:10 |
| pabelanger | ianw: maybe gentoo or opensuse? | 21:10 |
| pabelanger | we just brought gentoo online this week | 21:10 |
| pabelanger | but it is paused currently | 21:10 |
| *** armaan has joined #openstack-infra | 21:11 | |
| *** l4yerffej has joined #openstack-infra | 21:11 | |
| ianw | it does look like http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/20-iptables#n46 was applied on the host | 21:12 |
| *** bobh has joined #openstack-infra | 21:12 | |
| *** hongbin has quit IRC | 21:13 | |
| *** hongbin has joined #openstack-infra | 21:14 | |
| *** armaan has quit IRC | 21:15 | |
| fungi | i wonder if something is triggering a start/restart of netfilter-persistent (or equivalent) "service" in the chroot to apply the rules | 21:15 |
| fungi | chroot on its own won't do anything to protect against that | 21:16 |
| *** bobh has quit IRC | 21:16 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/20-iptables#n105 | 21:17 |
| pabelanger | might be it | 21:17 |
| pabelanger | as we just brought gentoo online this weekend | 21:17 |
| pabelanger | maybe not, as opensuse above does the same | 21:18 |
| clarkb | pabelanger: its different though | 21:18 |
| clarkb | opensuse is in a heredoc | 21:18 |
| clarkb | so its not executing that, ist saving it for later | 21:18 |
| ianw | yeah, i think that gentoo bit would be it | 21:18 |
| pabelanger | clarkb: ah, right | 21:19 |
| *** slaweq has joined #openstack-infra | 21:23 | |
| *** hongbin_ has joined #openstack-infra | 21:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove apply of iptables rules for gentoo https://review.openstack.org/561478 | 21:24 |
| ianw | ^ that seems like the simplest thing | 21:24 |
| *** bobh has joined #openstack-infra | 21:24 | |
| pabelanger | +2 | 21:24 |
| *** hongbin has quit IRC | 21:24 | |
| *** wolverineav has joined #openstack-infra | 21:24 | |
| pabelanger | prometheanfire: heads up ^ is breaking nodepool-builders | 21:25 |
| ianw | prometheanfire: ^^ i'm not sure how you want to run the rule apply on gentoo. something like netfilter-persistent, or a script to apply the rules | 21:25 |
| ianw | right, one yak shaved, now i can see the logs for the pip failures :) | 21:25 |
| fungi | i'm sure gentoo has _something_ which applies iptables rules at boot time. there's really no way around doing at least that | 21:27 |
| fungi | we just need to make sure whatever it is doesn't also trigger at image build time | 21:28 |
| *** bobh has quit IRC | 21:29 | |
| *** bobh has joined #openstack-infra | 21:32 | |
| *** bobh has quit IRC | 21:36 | |
| *** hongbin has joined #openstack-infra | 21:37 | |
| *** bobh has joined #openstack-infra | 21:37 | |
| *** hongbin_ has quit IRC | 21:38 | |
| *** efried has quit IRC | 21:40 | |
| *** salv-orl_ has quit IRC | 21:40 | |
| *** efried has joined #openstack-infra | 21:40 | |
| *** salv-orlando has joined #openstack-infra | 21:41 | |
| prometheanfire | ianw: didn't you see my other patch? The one enabling the iptables services for gentoo? | 21:41 |
| prometheanfire | 2018-04-15 21:42:14.588 | (0797 / 1372) | 21:42 |
| prometheanfire | taking forever :( | 21:42 |
| ianw | promethanfire: no, but ok, we still need to stop that apply of the rules on the builder however | 21:43 |
| prometheanfire | ianw: ok, so the problem is that the builder is applying the rules? I'm guessing opensuse is bad too then? | 21:44 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove installed packages before pip install https://review.openstack.org/561479 | 21:44 |
| ianw | promethanfire: no, opensuse is writing out a script to run on boot | 21:45 |
| ianw | pabelanger / fungi : ^^ to be tested, but that's my initial idea on this | 21:45 |
| prometheanfire | ianw: ok, so, maybe just enabling the services will work for gentoo then (or fixing it) since those will load the rules | 21:46 |
| *** hongbin_ has joined #openstack-infra | 21:51 | |
| ianw | promethanfire: you tell me, i'll believe you :) | 21:52 |
| *** bobh has quit IRC | 21:52 | |
| *** slaweq has quit IRC | 21:52 | |
| *** armaan has joined #openstack-infra | 21:52 | |
| *** hongbin has quit IRC | 21:53 | |
| *** VW has quit IRC | 22:08 | |
| *** VW has joined #openstack-infra | 22:08 | |
| *** bobh has joined #openstack-infra | 22:09 | |
| *** VW has quit IRC | 22:10 | |
| *** bobh has quit IRC | 22:14 | |
| *** sambetts_ has quit IRC | 22:15 | |
| *** sambetts_ has joined #openstack-infra | 22:16 | |
| *** bobh has joined #openstack-infra | 22:21 | |
| *** bobh has quit IRC | 22:25 | |
| *** jbadiapa has quit IRC | 22:26 | |
| prometheanfire | ianw: yes | 22:26 |
| *** rcernin has joined #openstack-infra | 22:26 | |
| *** jbadiapa has joined #openstack-infra | 22:26 | |
| *** bobh has joined #openstack-infra | 22:30 | |
| *** jbadiapa has quit IRC | 22:31 | |
| *** jbadiapa has joined #openstack-infra | 22:32 | |
| *** bobh has quit IRC | 22:35 | |
| *** r-daneel has joined #openstack-infra | 22:36 | |
| *** bobh has joined #openstack-infra | 22:40 | |
| *** rcernin has quit IRC | 22:45 | |
| *** rcernin has joined #openstack-infra | 22:46 | |
| *** bobh has quit IRC | 22:46 | |
| *** threestrands has joined #openstack-infra | 22:51 | |
| *** bobh has joined #openstack-infra | 22:52 | |
| *** bobh has quit IRC | 22:57 | |
| *** jamesmcarthur has quit IRC | 22:59 | |
| *** claudiub|2 has quit IRC | 23:03 | |
| *** lbragstad has quit IRC | 23:05 | |
| *** jamesmcarthur has joined #openstack-infra | 23:07 | |
| prometheanfire | 2018-04-15 23:10:18.737 | Couldn't parse 'functions-common: line 1301: rpm: command not found file /opt/cache/files/functions-common: line 1301: rpm: command not found /tmp/dib_build.Us7fIHQv/mnt/opt/git/openstack-dev/devstack/functions-common: line 1301: rpm: command not found' as a source repository | 23:11 |
| *** hongbin_ has quit IRC | 23:12 | |
| *** bobh has joined #openstack-infra | 23:13 | |
| prometheanfire | pabelanger: looks like devstack doesn't work on gentoo, as it was never suposed to, so build-image.sh should probably be modified to not hardcode the cache-devstack element | 23:15 |
| *** bobh has quit IRC | 23:17 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: include cache-devstack only if not gentoo https://review.openstack.org/561485 | 23:21 |
| *** bobh has joined #openstack-infra | 23:23 | |
| prometheanfire | pabelanger ianw ^ | 23:24 |
| prometheanfire | testing it now | 23:24 |
| *** stakeda has joined #openstack-infra | 23:28 | |
| *** bobh has quit IRC | 23:28 | |
| *** bobh has joined #openstack-infra | 23:32 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** bobh has quit IRC | 23:36 | |
| *** bobh has joined #openstack-infra | 23:42 | |
| pabelanger | prometheanfire: cache-devstack is more then that, it gets all our git repos and images onto the gentoo image. So, I think we'll want that, regardless of running devstack. | 23:45 |
| *** bobh has quit IRC | 23:47 | |
| prometheanfire | pabelanger: openstack-repos are still being cached | 23:47 |
| prometheanfire | something else is still pulling it in | 23:48 |
| pabelanger | prometheanfire: yah, it is a hard dependency on our builds | 23:49 |
| pabelanger | I don't think we can make it conditional for gentoo | 23:50 |
| prometheanfire | pabelanger: so I need to make devstack run on gentoo? | 23:50 |
| prometheanfire | that's totally NOT what I want to do | 23:51 |
| prometheanfire | I don't think any of the OSA jobs run devstack | 23:51 |
| *** jbadiapa has quit IRC | 23:51 | |
| prometheanfire | what else other than caching the repos needs doing? (since that's still being done) | 23:51 |
| ianw | there's room to consider what we're doing there; the caching of images is separate to the caching of the git trees | 23:52 |
| pabelanger | not run devstack, but ensure http://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/image_list.sh works | 23:52 |
| *** jbadiapa has joined #openstack-infra | 23:52 | |
| *** slaweq has joined #openstack-infra | 23:52 | |
| prometheanfire | run that after the image generation? | 23:54 |
| pabelanger | the cache-devstack element does it dynamically on each branch, so without seeing the build log, hard to say what is failing | 23:54 |
| *** bobh has joined #openstack-infra | 23:55 | |
| pabelanger | but, if you clone devstack to gentoo and run that script, should give an indication what is failing | 23:55 |
| prometheanfire | so... boot the generated image, clone devstack, run the script | 23:55 |
| prometheanfire | that the test procedure? | 23:55 |
| pabelanger | can you pastebin the log of failure above? I am just guessing that is the issue | 23:56 |
| prometheanfire | not sure it's still in backlog | 23:57 |
| prometheanfire | reverifying the cache of over 1k repos takes up a ton of log space | 23:57 |
| *** slaweq has quit IRC | 23:57 | |
| prometheanfire | https://gist.github.com/prometheanfire/1b4637e8fb2d92a8d237c407d704ceeb | 23:58 |
| prometheanfire | good thing I extended it :D | 23:58 |
| *** bobh has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!