| fungi | woohoo! expense reports caught up. also jenkins02 jjb resumed update completed after an additional 40 minutes | 00:00 |
|---|---|---|
| *** sridhar_ram1 has quit IRC | 00:01 | |
| crinkle | fungi: congratulations | 00:01 |
| nibalizer | jenkins you are so slow that expense reports take less time | 00:01 |
| nibalizer | think about that | 00:01 |
| fungi | indeed. my ability to get through mundane paperwork fills me with pride | 00:02 |
| nibalizer | fungi: will I see you next week? | 00:02 |
| nibalizer | in ftc? | 00:02 |
| fungi | indeedily-doodily | 00:02 |
| fungi | bindep experimental jobs are now adjusted to use constraints, so doing check experimental on the 40+ test changes in have open | 00:02 |
| *** jcooley_ has joined #openstack-infra | 00:03 | |
| *** harlowja_at_home has quit IRC | 00:03 | |
| *** harlowja_at_home has joined #openstack-infra | 00:04 | |
| *** asettle has joined #openstack-infra | 00:04 | |
| *** salv-orl_ has quit IRC | 00:05 | |
| nibalizer | crinkle: are you happy with 282573? | 00:09 |
| nibalizer | oh all voted on and processing | 00:09 |
| nibalizer | sweet | 00:09 |
| crinkle | heh | 00:09 |
| *** asettle has quit IRC | 00:09 | |
| crinkle | i'm so happy with it i've moved on to staring at zuul status | 00:10 |
| *** ybathia has quit IRC | 00:11 | |
| fungi | i'm just happy i'm finally able to wip and eventually abandon all these test changes i've submitted, now that the acl updates have all gone through | 00:13 |
| *** roxanagh_ has quit IRC | 00:13 | |
| fungi | doing batch commenting/voting on dozens of reviews in one shot from gertty is really fun | 00:13 |
| *** yamahata has quit IRC | 00:16 | |
| *** zz_dimtruck is now known as dimtruck | 00:17 | |
| *** ybathia has joined #openstack-infra | 00:19 | |
| *** ybathia has quit IRC | 00:19 | |
| *** Qiming has joined #openstack-infra | 00:23 | |
| mordred | crinkle: staring at zuul status is the best | 00:25 |
| mordred | crinkle: so - I was talking to russellb earlier today | 00:26 |
| *** rfolco has joined #openstack-infra | 00:26 | |
| *** rfolco has quit IRC | 00:26 | |
| mordred | crinkle: and he offered (sucker) to help us should we decide to shift to OVN backend for neutron | 00:26 |
| crinkle | mordred: cool! | 00:27 |
| mordred | crinkle: that's what I was thining - I like it when he helps with stuff | 00:27 |
| *** sarob_ has joined #openstack-infra | 00:28 | |
| mordred | crinkle: I also like backends that involve running fewer python agents resulting in considerably less rabbit messages that are all in service of features we don't use | 00:28 |
| *** harlowja_at_home has quit IRC | 00:28 | |
| crinkle | mordred: ah I was just about to ask about motivations for switching | 00:28 |
| mordred | for folks following along at home - with the ml2 agent, you get an increase in rabbit chatter proportional to number of compute nodes | 00:29 |
| mordred | even if you're using provider networks and the agents on the compute nodes are not actually doing anything | 00:29 |
| mordred | the more compute nodes you add, the more traffic there is in service of not doing anything | 00:29 |
| crinkle | do we have enough compute nodes for that to negatively affect our rabbit service? | 00:30 |
| mordred | we shoudl likely be fine with our number of compute nodes | 00:30 |
| *** _sarob has quit IRC | 00:31 | |
| mordred | however - given the volume of traffic nodepool likes to throw at clouds ... | 00:31 |
| mordred | it's also possible that the difference could matter. tough to say | 00:31 |
| mordred | the ovn architecture is definitely nicer - whether it would have a noticable impact on us ... | 00:32 |
| nibalizer | my feeling is I would like infracloud to put some nodes in the nodepool | 00:32 |
| mordred | ah | 00:33 |
| mordred | s/ah/yah/ | 00:33 |
| nibalizer | stopping to rearchitect the network is kinda like.... "but nodes!" | 00:33 |
| nibalizer | same with moving rabbit onto ssl | 00:33 |
| nibalizer | or moving the whole cloud to liberty | 00:33 |
| nibalizer | these things sound great! but I'd like to have nodepool run on it for a while so we get some return on investment | 00:34 |
| nibalizer | the lessons we'd learn in production would make theoretical conversations about performance easier to have anyways | 00:34 |
| *** logan- has quit IRC | 00:35 | |
| nibalizer | the hardware is getting moved anyways, so we'll have to rethink networking probably anyways | 00:35 |
| *** tzn has joined #openstack-infra | 00:36 | |
| *** woodster_ has quit IRC | 00:36 | |
| crinkle | I think it doesn't hurt to think about how to make the cloud better in some future time when someone has time to investigate it | 00:37 |
| *** sarob_ has quit IRC | 00:37 | |
| *** rguillebert has quit IRC | 00:37 | |
| mordred | yah | 00:37 |
| mordred | oh - I don't think we should rethink architectuure for the one that's about to have nodes | 00:38 |
| mordred | I think we should run nodes on it | 00:38 |
| mordred | otoh - I think yolanda has been trying to do repeated fresh installs in the other region ... so maybe doing ovn there for comparison/metrics would be fun - or maybe we just wait until the move | 00:39 |
| mordred | it's totally not urgent - more "a thing to think about as we think about things" | 00:39 |
| *** Swami has quit IRC | 00:39 | |
| crinkle | I'm avoiding doing much serious work until HPE figures out when and where these things are going | 00:39 |
| *** tzn has quit IRC | 00:40 | |
| *** logan- has joined #openstack-infra | 00:41 | |
| *** thorst has joined #openstack-infra | 00:41 | |
| *** verdurin has joined #openstack-infra | 00:43 | |
| mordred | crinkle: good call | 00:44 |
| *** grue_pm has quit IRC | 00:47 | |
| *** harlowja_at_home has joined #openstack-infra | 00:47 | |
| *** eil397 has quit IRC | 00:47 | |
| *** thorst has quit IRC | 00:48 | |
| *** thorst has joined #openstack-infra | 00:48 | |
| *** xarses has quit IRC | 00:48 | |
| *** grue_pm has joined #openstack-infra | 00:50 | |
| *** shashank_hegde has quit IRC | 00:52 | |
| *** eil397 has joined #openstack-infra | 00:52 | |
| *** thorst has quit IRC | 00:53 | |
| cody-somerville | Once we add this capacity, will we feel bad about iterating on it if it requires taking it down to say rearch network? | 00:55 |
| *** gyee has quit IRC | 00:56 | |
| *** |-paul-| has quit IRC | 00:56 | |
| *** yamamoto has joined #openstack-infra | 00:56 | |
| *** yamamoto has quit IRC | 01:01 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/gertty: Fixes bug "10000 revision not found" https://review.openstack.org/282585 | 01:02 |
| *** bpokorny has quit IRC | 01:03 | |
| *** eil397 has quit IRC | 01:03 | |
| *** gongysh has joined #openstack-infra | 01:04 | |
| cody-somerville | hmmm... my devstack with neutron run has hung on sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex :/ | 01:04 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/gertty: Fixes bug "10000 revision not found" https://review.openstack.org/282585 | 01:05 |
| pleia2 | that's because ovs is awful \o/ | 01:07 |
| pleia2 | but in all seriousness, it can be a bit fidgety | 01:08 |
| mordred | cody-somerville: nope - we should totally be able to reboot a whole cloud | 01:11 |
| cody-somerville | ah, I think maybe it is because I don't have kernel module loaded. | 01:11 |
| *** EricGonczer_ has quit IRC | 01:12 | |
| *** dims has quit IRC | 01:22 | |
| *** jpr has joined #openstack-infra | 01:22 | |
| *** dims has joined #openstack-infra | 01:22 | |
| *** sputnik13 has quit IRC | 01:24 | |
| *** jpr has quit IRC | 01:28 | |
| cody-somerville | lol, ipv6 in UIs are so ugly. | 01:29 |
| *** jpr has joined #openstack-infra | 01:29 | |
| *** yamamoto has joined #openstack-infra | 01:33 | |
| *** jpr has quit IRC | 01:33 | |
| *** dimtruck is now known as zz_dimtruck | 01:34 | |
| cody-somerville | So I'm going to hazard a guess something isn't quite working with my install... it seems to be in a loop creating interfaces. I suspect I'm going to find out what Linux's limit on those is. | 01:34 |
| mordred | cody-somerville: :) | 01:40 |
| *** smarcet has joined #openstack-infra | 01:45 | |
| *** roxanaghe has joined #openstack-infra | 01:45 | |
| *** apoorvad has quit IRC | 01:46 | |
| *** tiswanso has quit IRC | 01:46 | |
| *** tiswanso has joined #openstack-infra | 01:47 | |
| *** sdake has joined #openstack-infra | 01:48 | |
| *** smarcet has quit IRC | 01:50 | |
| *** roxanaghe has quit IRC | 01:52 | |
| *** thorst has joined #openstack-infra | 01:56 | |
| *** thorst has quit IRC | 01:57 | |
| *** thorst has joined #openstack-infra | 01:57 | |
| *** nwkarsten has joined #openstack-infra | 01:58 | |
| *** dims has quit IRC | 01:58 | |
| *** dims has joined #openstack-infra | 01:59 | |
| *** sdake has quit IRC | 02:00 | |
| *** nwkarsten has quit IRC | 02:03 | |
| *** thorst has quit IRC | 02:03 | |
| *** asselin_ has quit IRC | 02:03 | |
| *** ajmiller has quit IRC | 02:04 | |
| *** mtanino has quit IRC | 02:06 | |
| *** asettle has joined #openstack-infra | 02:10 | |
| *** yamamoto has quit IRC | 02:10 | |
| *** yamamoto has joined #openstack-infra | 02:14 | |
| *** yamamoto has quit IRC | 02:15 | |
| *** asettle has quit IRC | 02:15 | |
| *** roxanaghe has joined #openstack-infra | 02:20 | |
| *** Daisy has joined #openstack-infra | 02:22 | |
| *** andreykurilin__ has quit IRC | 02:23 | |
| *** roxanaghe has quit IRC | 02:23 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Move networking-odl rally job into check. https://review.openstack.org/282591 | 02:24 |
| *** Daisy has quit IRC | 02:27 | |
| *** tphummel has quit IRC | 02:40 | |
| *** claudiub has quit IRC | 02:42 | |
| *** _amrith_ is now known as amrith | 02:44 | |
| *** dims has quit IRC | 02:49 | |
| *** dims has joined #openstack-infra | 02:54 | |
| *** salv-orlando has joined #openstack-infra | 02:54 | |
| *** dims has quit IRC | 02:56 | |
| *** rhallisey has quit IRC | 02:57 | |
| *** salv-orlando has quit IRC | 02:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Add hpuswest public certificate to public hiera https://review.openstack.org/282573 | 03:06 |
| *** amrith is now known as _amrith_ | 03:09 | |
| *** asettle has joined #openstack-infra | 03:20 | |
| *** Wei_Liu has joined #openstack-infra | 03:21 | |
| *** harlowja_at_home has quit IRC | 03:22 | |
| *** asettle has quit IRC | 03:25 | |
| *** zz_dimtruck is now known as dimtruck | 03:25 | |
| crinkle | fungi: nibalizer cloud looks unbroken, thanks for your help! | 03:27 |
| *** tphummel has joined #openstack-infra | 03:29 | |
| *** Thelo has quit IRC | 03:31 | |
| *** EricGonczer_ has joined #openstack-infra | 03:33 | |
| *** EricGonczer_ has quit IRC | 03:38 | |
| *** asettle has joined #openstack-infra | 03:43 | |
| *** asettle has quit IRC | 03:48 | |
| *** wolverineav has joined #openstack-infra | 03:55 | |
| *** imcsk8 has quit IRC | 03:56 | |
| *** imcsk8 has joined #openstack-infra | 03:58 | |
| *** Liuqing has joined #openstack-infra | 04:04 | |
| *** jcooley_ has quit IRC | 04:04 | |
| *** asettle has joined #openstack-infra | 04:06 | |
| *** fedexo has joined #openstack-infra | 04:07 | |
| *** bhunter_ has quit IRC | 04:07 | |
| *** shashank_hegde has joined #openstack-infra | 04:07 | |
| *** tphummel has quit IRC | 04:08 | |
| *** asettle has quit IRC | 04:11 | |
| *** yolanda has quit IRC | 04:19 | |
| *** tiswanso has quit IRC | 04:26 | |
| *** jerryz has quit IRC | 04:30 | |
| *** esker has quit IRC | 04:33 | |
| *** esker has joined #openstack-infra | 04:43 | |
| *** Thelo has joined #openstack-infra | 04:45 | |
| nibalizer | crinkle: woot | 04:46 |
| *** salv-orlando has joined #openstack-infra | 04:48 | |
| *** xarses has joined #openstack-infra | 04:48 | |
| *** salv-orlando has quit IRC | 04:52 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add helper function to convert xml https://review.openstack.org/261620 | 04:52 |
| *** wolverineav has quit IRC | 04:59 | |
| *** Liuqing has quit IRC | 05:01 | |
| *** Liuqing has joined #openstack-infra | 05:02 | |
| *** vivekd has joined #openstack-infra | 05:12 | |
| *** chlong_ has joined #openstack-infra | 05:21 | |
| *** markvoelker has quit IRC | 05:24 | |
| *** _nadya_ has joined #openstack-infra | 05:24 | |
| *** yamahata has joined #openstack-infra | 05:25 | |
| *** dimtruck is now known as zz_dimtruck | 05:25 | |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Allow git review to set votes https://review.openstack.org/281504 | 05:29 |
| *** yamahata has quit IRC | 05:36 | |
| *** wolverineav has joined #openstack-infra | 05:41 | |
| *** wolverineav has quit IRC | 05:45 | |
| *** wolverineav has joined #openstack-infra | 05:46 | |
| *** yamahata has joined #openstack-infra | 05:49 | |
| *** Wei_Liu has quit IRC | 05:51 | |
| *** imcsk8 has quit IRC | 05:53 | |
| *** imcsk8 has joined #openstack-infra | 05:53 | |
| *** habib has joined #openstack-infra | 05:54 | |
| *** habib has quit IRC | 06:00 | |
| *** tzn has joined #openstack-infra | 06:01 | |
| *** tzn has quit IRC | 06:05 | |
| *** _nadya_ has quit IRC | 06:07 | |
| *** asettle has joined #openstack-infra | 06:09 | |
| *** asettle has quit IRC | 06:14 | |
| *** prometheanfire has quit IRC | 06:16 | |
| *** prometheanfire has joined #openstack-infra | 06:18 | |
| *** resker has joined #openstack-infra | 06:18 | |
| *** esker has quit IRC | 06:18 | |
| *** markvoelker has joined #openstack-infra | 06:24 | |
| *** markvoelker has quit IRC | 06:29 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 06:34 |
| *** fedexo has quit IRC | 06:38 | |
| *** max_lobur has joined #openstack-infra | 06:41 | |
| *** fedexo has joined #openstack-infra | 06:43 | |
| *** _nadya_ has joined #openstack-infra | 06:56 | |
| *** fedexo has quit IRC | 07:17 | |
| *** _nadya_ has quit IRC | 07:20 | |
| *** Liuqing has quit IRC | 07:25 | |
| *** [1]Thelo has joined #openstack-infra | 07:26 | |
| *** Thelo has quit IRC | 07:29 | |
| *** [1]Thelo is now known as Thelo | 07:29 | |
| *** mrmartin has joined #openstack-infra | 07:36 | |
| *** vivekd_ has joined #openstack-infra | 07:39 | |
| *** vivekd has quit IRC | 07:40 | |
| *** vivekd_ is now known as vivekd | 07:40 | |
| *** markvoelker has joined #openstack-infra | 07:40 | |
| *** markvoelker has quit IRC | 07:45 | |
| *** jtomasek has joined #openstack-infra | 07:47 | |
| *** mrmartin has quit IRC | 07:50 | |
| *** ociuhandu has joined #openstack-infra | 07:50 | |
| *** mrmartin has joined #openstack-infra | 07:58 | |
| *** waynr has quit IRC | 08:00 | |
| *** kozhukalov has joined #openstack-infra | 08:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:18 | |
| *** zeih has joined #openstack-infra | 08:22 | |
| *** salv-orlando has joined #openstack-infra | 08:25 | |
| *** asettle has joined #openstack-infra | 08:31 | |
| *** salv-orlando has quit IRC | 08:31 | |
| *** resker has quit IRC | 08:32 | |
| *** wolverineav has quit IRC | 08:33 | |
| *** wwwbukolaycom has quit IRC | 08:35 | |
| *** asettle has quit IRC | 08:36 | |
| *** waynr has joined #openstack-infra | 08:38 | |
| *** esker has joined #openstack-infra | 08:40 | |
| *** LinuxJedi has joined #openstack-infra | 09:05 | |
| *** vgridnev has joined #openstack-infra | 09:13 | |
| openstackgerrit | Martin Oemke proposed openstack-infra/project-config: Add python34 non-voting jobs for tacker projects https://review.openstack.org/282634 | 09:15 |
| *** Liuqing has joined #openstack-infra | 09:17 | |
| *** vgridnev_ has joined #openstack-infra | 09:20 | |
| *** vgridnev_ has quit IRC | 09:20 | |
| *** LinuxJedi has quit IRC | 09:21 | |
| *** vgridnev_ has joined #openstack-infra | 09:21 | |
| *** vgridnev has quit IRC | 09:23 | |
| *** zeih has quit IRC | 09:24 | |
| *** LinuxJedi has joined #openstack-infra | 09:28 | |
| *** yamamoto has joined #openstack-infra | 09:30 | |
| openstackgerrit | Merged openstack/requirements: Bumps os-win to 0.2.1 https://review.openstack.org/282351 | 09:30 |
| *** vgridnev_ has quit IRC | 09:32 | |
| *** vgridnev has joined #openstack-infra | 09:33 | |
| *** yamamoto has quit IRC | 09:34 | |
| *** tzn has joined #openstack-infra | 09:37 | |
| *** kzaitsev_mb has quit IRC | 09:38 | |
| *** Liuqing has quit IRC | 09:40 | |
| *** markvoelker has joined #openstack-infra | 09:41 | |
| *** tzn has quit IRC | 09:41 | |
| *** wolverineav has joined #openstack-infra | 09:45 | |
| *** markvoelker has quit IRC | 09:45 | |
| *** wolverineav has quit IRC | 09:51 | |
| *** rguillebert has joined #openstack-infra | 09:57 | |
| *** achanda has quit IRC | 10:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** esker has quit IRC | 10:32 | |
| *** yamamoto_ has joined #openstack-infra | 10:33 | |
| *** yamamoto has quit IRC | 10:33 | |
| *** esker has joined #openstack-infra | 10:38 | |
| *** openstackgerrit_ has quit IRC | 10:40 | |
| *** openstackgerrit_ has joined #openstack-infra | 10:41 | |
| *** vgridnev has quit IRC | 10:42 | |
| *** vgridnev has joined #openstack-infra | 10:42 | |
| *** kzaitsev_mb has quit IRC | 10:44 | |
| *** wolverineav has joined #openstack-infra | 10:48 | |
| *** openstackgerrit_ has quit IRC | 10:48 | |
| *** wolverineav has quit IRC | 10:52 | |
| *** asettle has joined #openstack-infra | 11:00 | |
| *** Liuqing has joined #openstack-infra | 11:01 | |
| *** _amrith_ is now known as amrith | 11:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:03 | |
| *** yamamoto_ has quit IRC | 11:05 | |
| *** asettle has quit IRC | 11:06 | |
| *** kzaitsev_mb has quit IRC | 11:16 | |
| *** yamamoto has joined #openstack-infra | 11:17 | |
| *** Jeffrey4l has joined #openstack-infra | 11:17 | |
| *** mrmartin has quit IRC | 11:17 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:29 | |
| *** markvoelker has joined #openstack-infra | 11:42 | |
| *** [HeOS] has joined #openstack-infra | 11:42 | |
| *** HeOS has quit IRC | 11:45 | |
| *** markvoelker has quit IRC | 11:47 | |
| *** kzaitsev_mb has quit IRC | 11:48 | |
| *** zeih has joined #openstack-infra | 11:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:49 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** kzaitsev_mb has quit IRC | 11:56 | |
| *** abregman|afk has joined #openstack-infra | 11:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:59 | |
| *** gongysh has quit IRC | 12:00 | |
| *** salv-orlando has joined #openstack-infra | 12:10 | |
| *** kzaitsev_mb has quit IRC | 12:14 | |
| *** salv-orlando has quit IRC | 12:17 | |
| *** sdake has joined #openstack-infra | 12:21 | |
| *** salv-orlando has joined #openstack-infra | 12:21 | |
| *** zeih has quit IRC | 12:21 | |
| *** gnuoy has quit IRC | 12:26 | |
| *** jamespage has quit IRC | 12:26 | |
| *** gnuoy has joined #openstack-infra | 12:26 | |
| *** jamespage has joined #openstack-infra | 12:26 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Git repos by merger using connection names https://review.openstack.org/282660 | 12:30 |
| *** Qiming has quit IRC | 12:30 | |
| *** Qiming has joined #openstack-infra | 12:31 | |
| *** salv-orlando has quit IRC | 12:32 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:32 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names https://review.openstack.org/282660 | 12:37 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Report job status 'QUEUED' for queued jobs https://review.openstack.org/272967 | 12:38 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Allow using numeric change number in addition to ID in Depends-On tag https://review.openstack.org/215553 | 12:38 |
| *** kzaitsev_mb has quit IRC | 12:40 | |
| *** Jeffrey4l has quit IRC | 12:41 | |
| *** markvoelker has joined #openstack-infra | 12:43 | |
| *** vgridnev has quit IRC | 12:45 | |
| *** vgridnev has joined #openstack-infra | 12:46 | |
| *** markvoelker has quit IRC | 12:47 | |
| *** wolverineav has joined #openstack-infra | 12:49 | |
| *** mrmartin has joined #openstack-infra | 12:49 | |
| *** wolverineav has quit IRC | 12:53 | |
| *** asettle has joined #openstack-infra | 12:53 | |
| *** Jeffrey4l has joined #openstack-infra | 12:54 | |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| *** asettle has quit IRC | 12:58 | |
| *** vgridnev has quit IRC | 13:05 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:05 | |
| *** vgridnev has joined #openstack-infra | 13:06 | |
| *** kzaitsev_mb has quit IRC | 13:12 | |
| *** tzn has joined #openstack-infra | 13:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:14 | |
| *** tzn has quit IRC | 13:18 | |
| *** hamzy has quit IRC | 13:24 | |
| *** kzaitsev_mb has quit IRC | 13:25 | |
| *** vivekd has quit IRC | 13:26 | |
| *** mikelk has joined #openstack-infra | 13:27 | |
| *** dims has joined #openstack-infra | 13:28 | |
| *** mikelk has quit IRC | 13:36 | |
| *** abregman|afk has quit IRC | 13:40 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:43 | |
| *** markvoelker has joined #openstack-infra | 13:44 | |
| *** mikelk has joined #openstack-infra | 13:46 | |
| *** markvoelker has quit IRC | 13:48 | |
| *** kzaitsev_mb has quit IRC | 13:49 | |
| *** wolverineav has joined #openstack-infra | 13:50 | |
| *** thorst has joined #openstack-infra | 13:51 | |
| *** anthonyper has quit IRC | 13:53 | |
| *** thorst has quit IRC | 13:54 | |
| *** anthonyper has joined #openstack-infra | 13:54 | |
| *** wolverineav has quit IRC | 13:54 | |
| *** thorst has joined #openstack-infra | 13:54 | |
| *** vivekd has joined #openstack-infra | 13:57 | |
| *** thorst has quit IRC | 13:58 | |
| *** vivekd has quit IRC | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** yamamoto has joined #openstack-infra | 14:04 | |
| *** yamamoto has quit IRC | 14:05 | |
| *** bgaifullin has joined #openstack-infra | 14:05 | |
| bgaifullin | folks please review & merge https://review.openstack.org/#/c/279526/ | 14:05 |
| openstackgerrit | Steven Dake proposed openstack-infra/irc-meetings: Kolla now splits meetings between US and APAC https://review.openstack.org/282681 | 14:10 |
| *** asettle has joined #openstack-infra | 14:12 | |
| openstackgerrit | Steven Dake proposed openstack-infra/irc-meetings: Kolla now splits meetings between US and APAC https://review.openstack.org/282681 | 14:15 |
| *** asettle has quit IRC | 14:18 | |
| *** dims has quit IRC | 14:18 | |
| *** smarcet has joined #openstack-infra | 14:18 | |
| *** vivekd has joined #openstack-infra | 14:18 | |
| *** smarcet has quit IRC | 14:19 | |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder: Depend on ifupdown in simple-init https://review.openstack.org/281843 | 14:19 |
| *** vivekd_ has joined #openstack-infra | 14:20 | |
| *** vivekd has quit IRC | 14:23 | |
| *** vivekd_ is now known as vivekd | 14:23 | |
| openstackgerrit | Steven Dake proposed openstack-infra/irc-meetings: Kolla now splits meetings between US and APAC https://review.openstack.org/282681 | 14:24 |
| *** tiswanso has joined #openstack-infra | 14:26 | |
| *** salv-orlando has joined #openstack-infra | 14:27 | |
| *** dims has joined #openstack-infra | 14:28 | |
| *** tiswanso has quit IRC | 14:28 | |
| *** tiswanso has joined #openstack-infra | 14:29 | |
| *** mrmartin has quit IRC | 14:34 | |
| openstackgerrit | Steven Dake proposed openstack-infra/irc-meetings: Kolla now splits meetings between US and APAC https://review.openstack.org/282681 | 14:37 |
| *** salv-orlando has quit IRC | 14:37 | |
| *** mrmartin has joined #openstack-infra | 14:47 | |
| *** wolverineav has joined #openstack-infra | 14:51 | |
| *** sdake has quit IRC | 14:53 | |
| *** mrmartin has quit IRC | 14:55 | |
| *** _nadya_ has joined #openstack-infra | 14:55 | |
| *** wolverineav has quit IRC | 14:55 | |
| *** baoli_ has quit IRC | 14:56 | |
| *** vgridnev has quit IRC | 14:57 | |
| *** vgridnev has joined #openstack-infra | 14:58 | |
| *** mrmartin has joined #openstack-infra | 15:00 | |
| *** tiswanso has quit IRC | 15:00 | |
| *** vgridnev has quit IRC | 15:00 | |
| *** vgridnev has joined #openstack-infra | 15:01 | |
| *** vgridnev has quit IRC | 15:02 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 15:04 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 15:05 |
| *** e0ne has joined #openstack-infra | 15:05 | |
| *** dingyichen has quit IRC | 15:07 | |
| *** zeih has joined #openstack-infra | 15:08 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| *** vgridnev has joined #openstack-infra | 15:09 | |
| *** dims has quit IRC | 15:13 | |
| *** yamamoto has quit IRC | 15:15 | |
| *** mrmartin has quit IRC | 15:18 | |
| *** vgridnev has quit IRC | 15:20 | |
| *** zeih has quit IRC | 15:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:23 | |
| mordred | fungi: so - not to be completely daft ... but the zuul cloner thing ... | 15:25 |
| mordred | fungi: why are things cloning from git.o.o in the first place? | 15:25 |
| mordred | fungi: shouldn't they be cloning from /opt/git ? | 15:25 |
| *** Qiming has quit IRC | 15:25 | |
| *** Liuqing has quit IRC | 15:26 | |
| fungi | mordred: look under the hood, z-c is supposed to actually clone from its configured cache if the cache is present, and then fetch more recent heads from the remote indicated | 15:30 |
| mordred | yes | 15:30 |
| mordred | but that's not what the output here says: https://jenkins05.openstack.org/job/gate-diskimage-builder-docs/482/console | 15:30 |
| mordred | fungi: if I run that same command myself on a devstack node (skipping the clone-map file) | 15:31 |
| fungi | mordred: though that brings up something i didn't check, which is that maybe the zuul-cloner replacement for gerrit-git-prep is missing an indication of where to look for the cache | 15:31 |
| mordred | INFO:zuul.Cloner:Creating repo openstack/diskimage-builder from cache file:///opt/git/openstack/diskimage-builder | 15:31 |
| mordred | fungi: the jenkins thing above has the --cache-dir argument, but does not mention cloning from cache | 15:31 |
| mordred | fungi: so, I think maybe something in the clone-map is causing it to skip cache dir? | 15:32 |
| fungi | are we missing the git cache, or is it malformed? not sure if you're trying from a held node? | 15:32 |
| mordred | I'm trying from a held node | 15:32 |
| mordred | and then copy and pasted the command from that jenkins run | 15:32 |
| fungi | i didn't think the clonemap did anything to alter the cache finding/using features of zc | 15:32 |
| mordred | possibly - but if I leave the clonemap file off (which I have to because I don't hav ea clonemap) | 15:33 |
| mordred | I get cached cloning | 15:33 |
| fungi | hrm. try adding a clonemap and see if it does like we're seeing in the job? | 15:33 |
| *** vgridnev has joined #openstack-infra | 15:33 | |
| fungi | though i looked at the traffic graphs for git.o.o and they didn't seem significantly higher like i would expect if a bunch of jobs suddenly started remotely cloning nova et al | 15:34 |
| mordred | fungi: k. I just tried with clonemap | 15:35 |
| mordred | and it also worked | 15:35 |
| mordred | fungi: weird right? | 15:35 |
| fungi | maybe it's an environment variable difference? | 15:35 |
| mordred | fungi: http://paste.openstack.org/show/487668/ | 15:35 |
| mordred | fungi: oh - that's a devstack-trusty | 15:36 |
| mordred | let me hold a bare-trusty | 15:36 |
| *** mikelk has quit IRC | 15:36 | |
| fungi | yeah, this has all been plaguing only bare-trusty | 15:36 |
| mordred | fungi: the cloning works for me on bare-trusty and does not attempt to clone from git | 15:39 |
| mordred | fungi: the cloning works for me on bare-trusty and does not attempt to clone from git.o.o | 15:39 |
| *** mikelk has joined #openstack-infra | 15:42 | |
| *** markvoelker has joined #openstack-infra | 15:44 | |
| *** vgridnev has quit IRC | 15:46 | |
| *** markvoelker has quit IRC | 15:49 | |
| *** wolverineav has joined #openstack-infra | 15:51 | |
| *** mikelk has quit IRC | 15:53 | |
| *** jcooley_ has joined #openstack-infra | 15:56 | |
| *** wolverineav has quit IRC | 15:56 | |
| *** amrith is now known as _amrith_ | 15:58 | |
| fungi | stranger and stranger | 15:59 |
| fungi | did you hold a bare-trusty node in rax-iad? that's where i've seen the bulk of failures | 15:59 |
| fungi | also, are we sure the issue hasn't gone away very recently? is there evidence we're still hitting it? | 16:00 |
| AJaeger | I had one this morning, fungi: http://logs.openstack.org/46/282646/1/check/gate-rally-python34/039262b/ | 16:01 |
| AJaeger | rax-iad | 16:01 |
| *** fedexo has joined #openstack-infra | 16:04 | |
| fungi | thanks for confirming, AJaeger | 16:04 |
| jeblair | fungi, mordred: if z-c clones from cache it should log at info level ("Creating repo %s from cache %s", | 16:05 |
| mordred | jeblair: yes. that's what I find to happen when I run it on the held nodes | 16:06 |
| *** mrmartin has joined #openstack-infra | 16:06 | |
| mordred | jeblair: the paste link in scrollback is my running it by hand on a held node | 16:06 |
| jeblair | mordred: ah cool; does it work on devstack-trusty? | 16:07 |
| mordred | jeblair: the jenkins link is a link to a job that failed due to cloning problems - and it shows that it is not cloning from cache | 16:07 |
| mordred | jeblair: it works on both devstack-trusty and bare-trusty when I run it by hand | 16:07 |
| jeblair | mordred: okay, so that's pointing to a zc problem then? | 16:07 |
| mordred | so I still do not know what is broken - but clearly something on the job above has been unhappy due to lack of cloning from cache | 16:07 |
| mordred | jeblair: | 16:08 |
| mordred | 2016-02-20 14:21:25.485 | INFO:zuul.Cloner:Creating repo openstack/diskimage-builder from upstream git://git.openstack.org/openstack/diskimage-builder | 16:08 |
| mordred | that line, in the job, is not what it should be | 16:08 |
| jeblair | ya, i see that :/ | 16:08 |
| jeblair | not os.path.exists(dest)): | 16:08 |
| jeblair | oh | 16:08 |
| jeblair | i think that's the problem | 16:08 |
| jeblair | weird | 16:08 |
| *** _amrith_ is now known as amrith | 16:08 | |
| *** fedexo has quit IRC | 16:08 | |
| jeblair | (because $WORKSPACE exists it's choosing to ignore the cache) | 16:09 |
| *** fedexo_ has joined #openstack-infra | 16:09 | |
| mordred | jeblair: so - I tried running with the same clonemap file that causes it try to clone to . | 16:09 |
| mordred | jeblair: and it worked | 16:09 |
| *** amrith is now known as _amrith_ | 16:11 | |
| *** _amrith_ is now known as amrith | 16:11 | |
| jeblair | mordred: i can't see how it could work? can you paste that session? | 16:15 |
| mordred | jeblair: yes | 16:15 |
| *** fedexo_ has quit IRC | 16:15 | |
| *** fedexo has joined #openstack-infra | 16:15 | |
| jeblair | cause now that i think about it, the tmpdir for requirements exists too | 16:15 |
| jeblair | so it should also fail that check and ignore cache | 16:16 |
| mordred | jeblair: http://paste.openstack.org/show/487670/ | 16:17 |
| jeblair | i wrote that line! | 16:17 |
| jeblair | i have no idea why i put that there. | 16:17 |
| jeblair | oh i do | 16:17 |
| *** fedexo_ has joined #openstack-infra | 16:18 | |
| jeblair | it's because that section of code is an explicit clone, so it's a (too-)simple check to see if it has already been cloned | 16:18 |
| jeblair | so basically, we need to replace that with something more like "is dest a valid git repo" rather than "does dest exist" | 16:18 |
| *** vivekd_ has joined #openstack-infra | 16:19 | |
| mordred | jeblair: ah - yes. that makes sense to me | 16:20 |
| *** vivekd has quit IRC | 16:20 | |
| mordred | jeblair: yah - like "does a git operation work in that repo" | 16:20 |
| *** vivekd_ is now known as vivekd | 16:20 | |
| jeblair | mordred: ah, i see the problem with your session -- the name of the dib repo is wrong: openstack-infra/diskimage-builder | 16:21 |
| jeblair | mordred: not -infra | 16:21 |
| *** fedexo has quit IRC | 16:21 | |
| jeblair | mordred: so zc is not finding entries for either of the projects in the clonemap file, and so is making new subdirs for them | 16:21 |
| mordred | hahaha | 16:22 |
| mordred | let me try again | 16:22 |
| jeblair | mordred: yeah, it's *shocking* the number of non-infra projects out there. i don't understand it. :) | 16:22 |
| mordred | jeblair: haha | 16:22 |
| mordred | jeblair: yes. fixing that in clonemap reproduces | 16:22 |
| jeblair | mordred: woot | 16:23 |
| *** tphummel has joined #openstack-infra | 16:23 | |
| mordred | so - that would certainly result in an increase of hitting the internet, which I would fully expect to increase incidence of failures | 16:23 |
| jeblair | mordred: well, that happened a *long* time ago i believe | 16:24 |
| *** amrith is now known as _amrith_ | 16:24 | |
| jeblair | mordred: so, inasmuch as rax-iad's internet may be broken, the fact that these jobs use the net more than necessary is a problem and will make them more prone to failure | 16:25 |
| jeblair | mordred: but these jobs switched to zc long before we started seeing the large number of issues on tuesday | 16:25 |
| *** Mmike has quit IRC | 16:25 | |
| *** Mmike has joined #openstack-infra | 16:25 | |
| AJaeger | jeblair: they synced "Tue Feb 16 08:11:32 2016 +0000" | 16:27 |
| AJaeger | Merge "Sync zuul-git-branch-prep-upper-constraints" | 16:27 |
| AJaeger | If we talk about the same thing... | 16:27 |
| AJaeger | Have to leave again... | 16:27 |
| mordred | jeblair: yah | 16:28 |
| mordred | jeblair: good point | 16:28 |
| jeblair | mordred: AJaeger makes a better one | 16:29 |
| jeblair | we really changed the job that was failing on the day it started failing and didn't notice? :) | 16:29 |
| jeblair | hrm | 16:30 |
| jeblair | AJaeger: that change should not have caused the kind of behavior change we're talking about | 16:30 |
| jeblair | that commit really only has any effect on changes to the requirements repo | 16:31 |
| jeblair | it doesn't change what happens for a nova change | 16:32 |
| *** Jeffrey4l has quit IRC | 16:32 | |
| mordred | ah | 16:32 |
| jeblair | (at least, it shouldn't... might be worth a look at an older nova run just to make sure the output is the same (no cache)) | 16:32 |
| *** sdake has joined #openstack-infra | 16:32 | |
| jeblair | mordred, lifeless, clarkb: are you aware of any weirdness with using multiprocessing and python testing? | 16:35 |
| jeblair | i'm looking at my change in https://review.openstack.org/282099 and wondering why tests are hanging after completion | 16:36 |
| *** aspiers has quit IRC | 16:36 | |
| mordred | jeblair: I am not | 16:38 |
| mordred | jeblair: yolanda tried looking in to that yesterday and was not able to solve that mystery | 16:38 |
| jeblair | oh, i wonder if the logging is not going to the testtools log capture thingy | 16:38 |
| *** aspiers has joined #openstack-infra | 16:44 | |
| *** aspiers has quit IRC | 16:44 | |
| *** vmbrasseur has quit IRC | 16:52 | |
| *** asettle has joined #openstack-infra | 16:52 | |
| *** vmbrasseur has joined #openstack-infra | 16:53 | |
| *** roxanaghe has joined #openstack-infra | 16:54 | |
| *** asettle has quit IRC | 16:57 | |
| *** hamzy has joined #openstack-infra | 16:57 | |
| *** aspiers has joined #openstack-infra | 16:59 | |
| *** rcernin has joined #openstack-infra | 16:59 | |
| *** mrmartin has quit IRC | 17:01 | |
| *** zeih has joined #openstack-infra | 17:04 | |
| *** _nadya_ has quit IRC | 17:06 | |
| *** salv-orlando has joined #openstack-infra | 17:06 | |
| *** asettle has joined #openstack-infra | 17:07 | |
| jeblair | i'm starting to think that using multiprocessing for this may be opening a whole can of worms :( | 17:08 |
| jeblair | i don't think it's going to be simple to solve, so we should probably shelve retries/timeout for now and focus instead on the cache fix. | 17:09 |
| *** salv-orlando has quit IRC | 17:11 | |
| *** asettle has quit IRC | 17:12 | |
| *** tphummel has quit IRC | 17:14 | |
| *** kzaitsev_mb has quit IRC | 17:16 | |
| *** armax has quit IRC | 17:16 | |
| *** _nadya_ has joined #openstack-infra | 17:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Cloner: use cache if dest exists https://review.openstack.org/282709 | 17:20 |
| jeblair | fungi, mordred: that is much simpler; would be good to get that on images soon. ^ | 17:20 |
| *** jpr has joined #openstack-infra | 17:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Cloner: use cache if dest exists https://review.openstack.org/282710 | 17:20 |
| *** [1]Thelo has joined #openstack-infra | 17:27 | |
| *** Thelo has quit IRC | 17:30 | |
| *** [1]Thelo is now known as Thelo | 17:30 | |
| sdake | jeblair clarkb I'd like to reopen the discussion about special snowflake VMs with 16gb of ram | 17:31 |
| sdake | is there a technical limitiation to doing the job there? | 17:31 |
| sdake | or a cost thing? | 17:31 |
| sdake | jeblair clarkb do you suggest i start a thread on the mailing list? | 17:32 |
| sdake | our gating is severely limited by 8gb ram, and people are coming up with "hey lets run CI in my house" type solutions | 17:33 |
| sdake | which well intended is not the right approach :) | 17:33 |
| fungi | the main cost is that we're attempting to reduce the number of different sorts of single-use virtual machines we're running for jobs (for example, current push to get devstack jobs and units tests/static analyzers all running on one platform). also having different virtual machines use different amounts of ram makes quota calculations a *lot* harder | 17:37 |
| *** jpr has quit IRC | 17:37 | |
| sdake | fungi so lets separte those two into two different things becuase I thin kthey aredifferent problems | 17:38 |
| sdake | by single use vms, you mean you want everything running on ubuntu insteand of ubuntu+centos? | 17:39 |
| sdake | (for devstack jobs which kolla jobs most closely match) | 17:39 |
| fungi | we'd likely need to put them in a separate tenant on the clouds where we run them if we did that, with its own quota, and then either their availability would be constrained (causing changes needing them to wait waaay longer to report under heavy ci load) or they'd be needlessly wasting available quota | 17:39 |
| fungi | sdake: no, i was referring to our current effort of collapsing our bare-trusty and devstack-trusty workers to a single ubuntu-trusty type | 17:40 |
| fungi | for example | 17:40 |
| sdake | oh right that toally makes sense | 17:40 |
| sdake | we will totally comply with anything you need ther e in kolla ;) | 17:40 |
| sdake | we do need centos and ubuntu tho, because centos won't build on ubuntu | 17:40 |
| sdake | so centos has to build on centos | 17:40 |
| fungi | but the real challenge, i think, is having a mix of ram sizes in one tenant | 17:40 |
| sdake | and ubuntu on ubuntu | 17:40 |
| fungi | chroot doesn't work on ubuntu any more? | 17:41 |
| sdake | its a kernel issue | 17:41 |
| fungi | ahh | 17:41 |
| sdake | soemthing with cap_set_perm or something | 17:41 |
| fungi | just need a newer kernel? | 17:41 |
| sdake | nope I am nto sure how to rectify, we are using latest kernel | 17:41 |
| sdake | basically centos doesn't build on ubuntu because rpm does some selinux stuff (I think) that is incompatible with ubuntu | 17:41 |
| sdake | even though we turn selinux off | 17:42 |
| sdake | I think ubuntu builds on centos, but the results are not trustworthy because containers need to run on the base architecture they were built for | 17:42 |
| sdake | i.e. /usr/include needs to match | 17:42 |
| sdake | that includes the kernel headers | 17:42 |
| sdake | whcih are obviously different :) | 17:43 |
| sdake | glibc is different | 17:43 |
| sdake | but it seems like we are solid on that front, your not suggesting to get rid of centos, so we are set there ;) | 17:43 |
| sdake | the multi-tenant idea, is that a capacity issue from engineering, or a "cant be done because the cloud providers wont allow it" issue? | 17:43 |
| *** _nadya_ has quit IRC | 17:44 | |
| sdake | I think besides nova, kolla isprobably #2 in terms of "abuse" of infra ;) | 17:44 |
| *** dims has joined #openstack-infra | 17:45 | |
| *** jpr has joined #openstack-infra | 17:45 | |
| sdake | its just a fact of how our software operates | 17:45 |
| fungi | just trying to think through where the challenges will be in that. obviously more input from other infra team members will be useful | 17:45 |
| *** markvoelker has joined #openstack-infra | 17:45 | |
| sdake | fungi i'll start a thread on the ml then? | 17:45 |
| fungi | but like me, most are probably packing/travelling for our infra-cloud mid-cycle | 17:45 |
| sdake | we really need more memory | 17:45 |
| fungi | sure, just expect some lag in responses | 17:45 |
| sdake | i'm good with lag | 17:46 |
| clarkb | is 2x8gb not sufficient? | 17:46 |
| fungi | given most of us are sequestered for the coming week | 17:46 |
| sdake | even 12gb would do | 17:46 |
| sdake | clarkb out compute kit takes 7.5gb of ram to deploy | 17:46 |
| fungi | i mean, we build them with 16gb virtual memory already | 17:46 |
| fungi | just 8gb of that is swap | 17:46 |
| sdake | out/our | 17:46 |
| clarkb | sdake: that seems excessive | 17:47 |
| sdake | the swap times out our gates unfortunately | 17:47 |
| sdake | clarkb blame python ;) | 17:47 |
| fungi | also multi-node workers have two machines with 8gb of actual ram | 17:47 |
| clarkb | so my first suggestion is to address the memory use directly forst | 17:47 |
| clarkb | magnum has done this to a degree | 17:47 |
| sdake | we have, by removing all services that are not part of the compute kit | 17:47 |
| clarkb | neutron has had to in the past and fixed real bugs | 17:47 |
| clarkb | etc | 17:47 |
| sdake | but we prefer to test all services in the gate | 17:47 |
| clarkb | you dont need 7.5gb of ram to run neutron and nova compute agents | 17:48 |
| sdake | the 7.5gb of ram is used by openstack services running together | 17:48 |
| sdake | glance+keystone_mariadb+rabbitmq+neutron+nvoa+agents | 17:48 |
| clarkb | anyways we have 2x8gb tpday | 17:48 |
| clarkb | so that is an option available to you right now | 17:48 |
| sdake | we also limited the number of worker threads to 1 | 17:49 |
| sdake | what is 2x8gb? | 17:49 |
| sdake | 2 nodes, with 8gb each? | 17:49 |
| *** _nadya_ has joined #openstack-infra | 17:49 | |
| clarkb | yes | 17:49 |
| sdake | we want to deploy the compute kit on 3 nodes for ha purpose testing | 17:49 |
| sdake | each compute kit is a complete replica of the other | 17:49 |
| clarkb | ok thag is a completely separatr problem so lrts not complicatr matters... | 17:49 |
| *** markvoelker has quit IRC | 17:49 | |
| clarkb | does 2x8gb help? | 17:50 |
| sdake | no, becaue we can't run different services on different nodes | 17:50 |
| sdake | a control node runs all services, not a subset | 17:50 |
| sdake | atleast that is what we want to test, because that is how operators use it | 17:50 |
| sdake | and down the road we want the control node to run on 3 vms and interact | 17:51 |
| sdake | this takes even more then 8gb of ram | 17:51 |
| fungi | you mean it takes more than 8gb of ram for each of the 3 workers? | 17:52 |
| sdake | we just got to the point we can launch a dead chicken test (launch 1 vm of very small size + minimal compute kit) | 17:52 |
| fungi | or you mean you're trying to run 3 virtual machines on one virtual machine? | 17:52 |
| *** wolverineav has joined #openstack-infra | 17:52 | |
| sdake | fungi we want to run 3 control nodes in the future | 17:52 |
| sdake | but for the moment i'd take one control node that was a complete deployment | 17:53 |
| fungi | so a three-node worker with one control node on each then | 17:53 |
| *** zeih has quit IRC | 17:53 | |
| sdake | atm we are turning off over half our services to run | 17:53 |
| sdake | fungi ack | 17:53 |
| clarkb | sdake but not one control node and one compue node? | 17:53 |
| sdake | clarkb actually thats a workable idea | 17:53 |
| clarkb | ok then 2x8gb would work? | 17:53 |
| sdake | that would be a step in the right direction, i'll take that back to the team | 17:54 |
| sdake | it would work for the non-ha case for sure | 17:54 |
| sdake | not sure we can get all the services deployed in 8gb ram | 17:54 |
| sdake | but clearly it would help kolla if we expanded gating past one node, so maybe that is he next step | 17:54 |
| sdake | our control and compute and storage are separate models | 17:55 |
| sdake | that can be deployed to independent nodes | 17:55 |
| sdake | atm we deploy them all to one node | 17:55 |
| *** rcernin has quit IRC | 17:55 | |
| sdake | so control on one, compute on another, would let us run more tests | 17:55 |
| sdake | still nto enough ram to deploy the big tent ;) | 17:56 |
| sdake | but we can work from there | 17:56 |
| fungi | jeblair: interesting gertty error, when reviewing 282709 if i try to follow the link on its change-id commit message header (presumably because it's shared with 282710) i get a pop-up dialog which says "Error: Multiple rows were found for one()" | 17:56 |
| fungi | mordred: did you intend to review 282709 (the master branch change) rather than 282710 (for feature/zuulv3)? | 17:57 |
| sdake | clarkb do you ahve a pointer to this 2x8 setup (is anyone else doing that?) | 17:57 |
| *** wolverineav has quit IRC | 17:57 | |
| clarkb | really the issue with different sizes of VMs is csheduling across heterogenous flavors across multiple regions in multiple clouds. on ovh we have one flavor we can use, in other region the ratio between ram and disk and cpu is all different and changes between flavors which means while you might be able to boot 100x8gb in this region you can only do 25x16gb but in another region you can do 100x8 and | 17:57 |
| clarkb | 50x8 | 17:57 |
| mordred | fungi: uh | 17:57 |
| clarkb | it makes the problem space much more complicated | 17:57 |
| mordred | fungi: yup | 17:57 |
| mordred | fungi: done | 17:57 |
| clarkb | sdake: yes many things are using it, the grenade and tempest mulyinode jobs for example | 17:58 |
| sdake | clarkb just to be clear on that, ovh only has an 8gb flavor option, not a 16gb option? | 17:58 |
| mordred | for us | 17:58 |
| clarkb | sdake: for our tenant | 17:58 |
| mordred | we have a custom flavor there | 17:58 |
| mordred | sdake: getting similar flavors on 5-6 different clouds turns out to be very hard | 17:58 |
| sdake | mordred yup sounds hard | 17:59 |
| fungi | standardization has helped us a _lot_ on being able to spread across providers | 17:59 |
| sdake | for some reason we can't run our tests on ovh at all at the moment | 17:59 |
| *** rcernin has joined #openstack-infra | 18:00 | |
| fungi | likely due to either lack of swap or only being able to use a swapfile | 18:00 |
| sdake | our test is deploy + boot vm + test in active state | 18:00 |
| sdake | i'd like to run rally and tempest as well | 18:00 |
| fungi | or possibly not enough local disk? we only have effectively ~70gb of rootfs there after preallocating the swapfile i think | 18:00 |
| *** kozhukalov has quit IRC | 18:00 | |
| openstackgerrit | Merged openstack-infra/zuul: Cloner: use cache if dest exists https://review.openstack.org/282709 | 18:01 |
| SamYaple | o/ | 18:02 |
| SamYaple | oh we are good on RAM now | 18:02 |
| SamYaple | after tweaking configs, we have ~4GB left | 18:02 |
| *** bgaifullin has quit IRC | 18:02 | |
| SamYaple | 3 after launching a cirros image | 18:02 |
| *** bgaifullin has joined #openstack-infra | 18:03 | |
| *** bgaifullin has quit IRC | 18:03 | |
| clarkb | SamYaple: cirros boots in 64mb | 18:03 |
| SamYaple | yea clarkb but the default flavors are 1gb | 18:03 |
| SamYaple | my point was we have plenty to go around atm | 18:04 |
| SamYaple | we dropped a few services | 18:04 |
| mordred | woot | 18:05 |
| sdake | yes but the reality is I want to test all deploy not just compute kit | 18:05 |
| SamYaple | i am doubtful thats going to happen in any meaningful way to be honest | 18:05 |
| SamYaple | not wit hcurrent resources | 18:05 |
| sdake | that is what this conversation is about :) | 18:06 |
| sdake | fungi kicked out a 2nd tenant idea with 16gb vms | 18:06 |
| *** dims has quit IRC | 18:06 | |
| fungi | that was a strawman, i doubt we could get consensus on it | 18:06 |
| fungi | and i'm not particularly fond of the idea myself | 18:06 |
| sdake | fungi atm just looking for options | 18:06 |
| sdake | i like control on one node compute on another | 18:07 |
| sdake | that might get us rollign with more services | 18:07 |
| clarkb | basically the problem thst needs solving is multidimensional scheduling | 18:07 |
| SamYaple | it will not sdake | 18:07 |
| sdake | so its on us to try that out and see how well that works | 18:07 |
| SamYaple | weve already talked about that | 18:07 |
| fungi | was mostly using it as an example of the sorts of inefficient workarounds we'd have to employ to deal with heterogenous workers | 18:07 |
| clarkb | hardcoding a separate tenant is one way to do that | 18:07 |
| *** e0ne has quit IRC | 18:07 | |
| SamYaple | if youll recall the conversation, the compute node doesnt use alot of resources | 18:07 |
| *** ashleighfarnham has joined #openstack-infra | 18:07 | |
| SamYaple | ill be honest, i think we have room to grow in the gate until at least start of newton | 18:07 |
| sdake | yes this is post newton work | 18:08 |
| SamYaple | we definetely have room to setup the multinode gates properly now that RAM isn't an issue for the moment | 18:08 |
| SamYaple | im not sure we will even need to go 16gb in newton | 18:08 |
| sdake | i wanted to start the conversation now so folks can start thinkingabout it | 18:08 |
| sdake | yes if you read scrollback i think 12gb would do | 18:08 |
| SamYaple | I think 8gb is fine for now | 18:08 |
| SamYaple | even into newton | 18:09 |
| sdake | but there are 40 services in big tent | 18:09 |
| sdake | down the road we will deploy all of them | 18:09 |
| *** woodster_ has joined #openstack-infra | 18:09 | |
| clarkb | you dont have to test them all together in one go... | 18:09 |
| SamYaple | yea | 18:09 |
| SamYaple | exactly | 18:09 |
| sdake | clarkb i proposed that idea | 18:09 |
| clarkb | in fact doing so is probably a bad idea for other reasons | 18:09 |
| sdake | SamYaple thinks running multiple gates to subtest sections is hard on infra resources | 18:09 |
| sdake | clarkb yoru thoughts? | 18:09 |
| fungi | that sounds remarkably ambitious. i don't think we have any devstack jobs which try to deploy the entirety of the big tent in one shot, nor would i expect that to be viable on constrained workers | 18:09 |
| SamYaple | sdake: no no i said i dont think we want a gate for ever conviebale combonation | 18:10 |
| clarkb | sdake: I think it is a terrible idea because it unfocuses the test and makes it harder to debug | 18:10 |
| fungi | i mean, that was devstack's function once upon a time, but once we got past about 7(?) official server projects we started having to turn some of them off to test others | 18:10 |
| sdake | clarkb agree, but didnt' you just suggest it :) | 18:11 |
| clarkb | yup | 18:11 |
| clarkb | sdake: I suggested not testing everything at once | 18:11 |
| clarkb | testing evrything at once is not a good idea | 18:11 |
| sdake | so a gate for heat addon, gate for murano addon etc? | 18:12 |
| clarkb | not necessarily | 18:12 |
| clarkb | one job for compute, one job for storage, one job for orchestration or whatever | 18:12 |
| clarkb | you dont have to be on one extreme or the other | 18:12 |
| sdake | our orchestratoin (ansible) is really light weight | 18:12 |
| clarkb | sdake: you just mentioned heat... | 18:13 |
| sdake | all the openstack services jam up the ram usage | 18:13 |
| *** _nadya_ has quit IRC | 18:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:13 | |
| sdake | oh right | 18:13 |
| sdake | yes I did propose what you just said, and was asking forfeedback on "if that would kill infra" :) | 18:13 |
| fungi | no more than having a ceilometer-specific vs designate-specific devstack/tempest job does | 18:14 |
| sdake | one of the issues is heat has become a core dependency for many services, and is a huge memory user | 18:14 |
| sdake | fungi so sounds like infra is designed to handle that situation | 18:14 |
| sdake | i think if we use the 2x8 and put compute on one and control on another | 18:15 |
| sdake | we can selectivley ad d gate jobs for each of the additional big tent services | 18:15 |
| SamYaple | waht services is heat a core dependacy of? | 18:15 |
| sdake | but we could potentiallly end up with 40 jobs per distro | 18:15 |
| sdake | magnum for example | 18:16 |
| SamYaple | magnum and murano are the only ones i can think of | 18:16 |
| sdake | everyone uses heat because they dont want to deal with talking to the opentack apis directly | 18:16 |
| kzaitsev_ws | true, murano currently heavily depends on heat =) | 18:16 |
| sdake | that is what heat's purpose is | 18:16 |
| clarkb | which us why it needs so much memory | 18:17 |
| clarkb | etlr | 18:17 |
| clarkb | gah | 18:17 |
| clarkb | why it is odd it needs so much memory | 18:17 |
| sdake | we are nto sure hwy heat uses so muc memkory | 18:17 |
| sdake | but it does | 18:17 |
| sdake | even in an idle state | 18:17 |
| sdake | SamYaple if we add heat and magnum, the memory usage is good to go with 8gb vms? | 18:18 |
| sdake | (if we put compute on a separate vm) | 18:18 |
| sdake | you mentioned it wont work because of internap? | 18:18 |
| clarkb | magnum runs their tests in a single 8gb vm fwiw | 18:19 |
| sdake | i just want to avoid the requirement for third party ci's run out of peoples houses | 18:19 |
| sdake | because then they could never be voting | 18:19 |
| sdake | clarkb magnum just removed a whole slew of testing because of memory constraints and gate timeout | 18:19 |
| SamYaple | no i said internap times out alot. im still collecting the info infra needs to fix that issue | 18:20 |
| sdake | I think they would benefit from a 2x8 config as well | 18:20 |
| *** tphummel has joined #openstack-infra | 18:20 | |
| clarkb | they did? | 18:20 |
| clarkb | we addressed a large portion of the issue and as far as I know the functional test is still running | 18:20 |
| *** mrmartin has joined #openstack-infra | 18:20 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support default string values for variables https://review.openstack.org/180117 | 18:21 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Allow id's for different templates to follow same naming https://review.openstack.org/282723 | 18:21 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Document template variable default values https://review.openstack.org/282724 | 18:21 |
| clarkb | protip dont use 2gb nested VMs | 18:21 |
| sdake | clarkb i may have dated information | 18:21 |
| kzaitsev_ws | a bit distracted, but if you have any questions about murano, including it's gates — feel free to ping me =) | 18:22 |
| sdake | clarkb i am just running with what hits the mailing list was "lets remove all the testss" :) | 18:22 |
| sdake | clarkb I think if we setup 2x8 and compute on one and control on another we can get alot more deployment going | 18:23 |
| sdake | coupled withbreaking out by service type we may be able to fit in 8gb vms | 18:23 |
| sdake | thanks for the coaching ;) | 18:23 |
| sdake | sounds like infra is good with that course of action | 18:23 |
| clarkb | the issues were memory related, first the extra room for VMs was about 1.25gb but the job booted 2x2gb | 18:24 |
| *** dims has joined #openstack-infra | 18:24 | |
| sdake | oh lol ;) | 18:24 |
| clarkb | this meant swap was super important but some regions didnt have swap | 18:24 |
| clarkb | so we addressed both issues, use smallwr VMs and add swap | 18:24 |
| sdake | clarkb haven't been involved with magnum forabout 4 months, so not up to date | 18:24 |
| clarkb | I dont know that this is 100% working yet but aiui it is mich more reliable | 18:25 |
| sdake | anyway the point was magnum uses heat, which is what i was answering :) | 18:25 |
| sdake | and heat uses the compute kit | 18:25 |
| sdake | i'm surprised heat isn't part of compute kit tbh | 18:25 |
| sdake | its becoming more and more mandatory | 18:25 |
| sdake | thanks folks for the feedback, I think we have a plan of action here to use infra resources more effectively with 2x8 vms | 18:26 |
| sdake | which we are not doing at present | 18:26 |
| sdake | if we follow that course, the natural extension forus is 4x8vms (3 control, 1 compute) | 18:27 |
| sdake | our softwre sort of works with 2 control 1 compute but not ideal :) | 18:27 |
| *** Sukhdev has joined #openstack-infra | 18:29 | |
| *** kzaitsev_mb has quit IRC | 18:30 | |
| clarkb | health dashboard only knows about two magnum functional tests but both passed, with more data should be able to see how reliable that setup is | 18:38 |
| *** ashleighfarnham has quit IRC | 18:38 | |
| *** jpr has quit IRC | 18:39 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/irc-meetings: Add rpm-packaging team meeting https://review.openstack.org/279950 | 18:39 |
| *** _nadya_ has joined #openstack-infra | 18:40 | |
| *** jpr has joined #openstack-infra | 18:42 | |
| *** dims has quit IRC | 18:43 | |
| *** dims has joined #openstack-infra | 18:43 | |
| *** vivekd_ has joined #openstack-infra | 18:44 | |
| *** markvoelker has joined #openstack-infra | 18:46 | |
| *** vivekd has quit IRC | 18:47 | |
| *** sdake has quit IRC | 18:47 | |
| *** vivekd_ is now known as vivekd | 18:48 | |
| *** asettle has joined #openstack-infra | 18:50 | |
| *** markvoelker has quit IRC | 18:50 | |
| *** roxanaghe has quit IRC | 18:51 | |
| fungi | okay, travel for infra-cloud sprint begins now. off to overnight in norfolk so i can catch an early flight out tomorrow | 18:54 |
| fungi | see lots of you there! | 18:55 |
| *** asettle has quit IRC | 18:55 | |
| clarkb | fungi see you tomorrow | 18:56 |
| *** nwkarsten has joined #openstack-infra | 18:56 | |
| *** mrmartin has quit IRC | 19:03 | |
| *** EricGonczer_ has joined #openstack-infra | 19:04 | |
| *** jpr has quit IRC | 19:07 | |
| *** wolverineav has joined #openstack-infra | 19:12 | |
| *** achanda has joined #openstack-infra | 19:12 | |
| *** tphummel has quit IRC | 19:16 | |
| *** tphummel has joined #openstack-infra | 19:16 | |
| *** tphummel has quit IRC | 19:18 | |
| *** wolverineav has quit IRC | 19:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:21 | |
| *** wolverineav has joined #openstack-infra | 19:21 | |
| *** ashleighfarnham has joined #openstack-infra | 19:21 | |
| *** wolverineav has quit IRC | 19:22 | |
| nibalizer | see you fungi. travel safe | 19:27 |
| *** kzaitsev_mb has quit IRC | 19:31 | |
| *** dims has quit IRC | 19:33 | |
| *** mrmartin has joined #openstack-infra | 19:34 | |
| *** _diana_ has quit IRC | 19:37 | |
| *** tphummel has joined #openstack-infra | 19:39 | |
| bkero | Bye | 19:42 |
| *** ttx has quit IRC | 19:42 | |
| *** nwkarsten has quit IRC | 19:45 | |
| *** ttx has joined #openstack-infra | 19:45 | |
| *** nwkarsten has joined #openstack-infra | 19:46 | |
| *** wolverineav has joined #openstack-infra | 19:46 | |
| *** kmartin has quit IRC | 19:47 | |
| *** _diana_ has joined #openstack-infra | 19:49 | |
| *** nwkarsten has quit IRC | 19:50 | |
| *** ashleighfarnham has quit IRC | 19:53 | |
| *** _nadya_ has quit IRC | 19:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:01 | |
| *** Sukhdev has quit IRC | 20:03 | |
| *** mrmartin has quit IRC | 20:04 | |
| *** kzaitsev_mb has quit IRC | 20:16 | |
| *** woodster_ has quit IRC | 20:16 | |
| *** nwkarsten has joined #openstack-infra | 20:24 | |
| *** jpr has joined #openstack-infra | 20:28 | |
| *** dave-mccowan has joined #openstack-infra | 20:28 | |
| *** wolverineav has quit IRC | 20:33 | |
| *** dims has joined #openstack-infra | 20:38 | |
| *** |-paul-| has joined #openstack-infra | 20:39 | |
| *** BobBall_AWOL has joined #openstack-infra | 20:40 | |
| *** wolverineav has joined #openstack-infra | 20:40 | |
| *** jpr has quit IRC | 20:44 | |
| *** salv-orlando has joined #openstack-infra | 20:44 | |
| *** vivekd_ has joined #openstack-infra | 20:45 | |
| *** markvoelker has joined #openstack-infra | 20:47 | |
| *** vivekd has quit IRC | 20:47 | |
| *** Roamer` has quit IRC | 20:48 | |
| *** Roamer` has joined #openstack-infra | 20:49 | |
| *** salv-orlando has quit IRC | 20:50 | |
| *** sdake has joined #openstack-infra | 20:51 | |
| *** asettle has joined #openstack-infra | 20:51 | |
| *** markvoelker has quit IRC | 20:51 | |
| *** vivekd_ has quit IRC | 20:51 | |
| *** dims has quit IRC | 20:53 | |
| *** dims has joined #openstack-infra | 20:54 | |
| *** woodster_ has joined #openstack-infra | 20:54 | |
| *** rcernin has quit IRC | 20:56 | |
| *** asettle has quit IRC | 20:56 | |
| *** tojuvone_ has quit IRC | 20:58 | |
| *** tojuvone_ has joined #openstack-infra | 20:58 | |
| *** fedexo_ has quit IRC | 21:00 | |
| *** EricGonczer_ has quit IRC | 21:06 | |
| *** salv-orlando has joined #openstack-infra | 21:07 | |
| *** EricGonczer_ has joined #openstack-infra | 21:07 | |
| *** BobBall_AWOL has quit IRC | 21:09 | |
| *** zz_dimtruck is now known as dimtruck | 21:12 | |
| *** nwkarsten has quit IRC | 21:13 | |
| *** EricGonczer_ has quit IRC | 21:13 | |
| *** nwkarsten has joined #openstack-infra | 21:13 | |
| *** EricGonczer_ has joined #openstack-infra | 21:14 | |
| *** |-paul-| has quit IRC | 21:14 | |
| *** salv-orlando has quit IRC | 21:14 | |
| *** tzn has joined #openstack-infra | 21:17 | |
| *** nwkarsten has quit IRC | 21:18 | |
| *** sdake has quit IRC | 21:19 | |
| *** sdake has joined #openstack-infra | 21:24 | |
| *** dims has quit IRC | 21:29 | |
| *** rcernin has joined #openstack-infra | 21:30 | |
| *** leakypipes has quit IRC | 21:45 | |
| *** dimtruck is now known as zz_dimtruck | 21:49 | |
| *** tzn has quit IRC | 21:50 | |
| *** sdake has quit IRC | 21:50 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: fix building on gentoo hosts https://review.openstack.org/282193 | 21:52 |
| *** armax has joined #openstack-infra | 21:52 | |
| openstackgerrit | Merged openstack/requirements: Bump oslo.utils to >= 3.5 to get to_utf8() https://review.openstack.org/281807 | 21:53 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 21:53 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 21:55 |
| prometheanfire | SpamapS: thanks :D | 21:55 |
| *** achanda has quit IRC | 21:56 | |
| *** eliqiao_ has quit IRC | 22:02 | |
| *** eliqiao_ has joined #openstack-infra | 22:05 | |
| *** zeih has joined #openstack-infra | 22:06 | |
| *** wolverineav has quit IRC | 22:07 | |
| *** wolverineav has joined #openstack-infra | 22:08 | |
| *** armax has quit IRC | 22:14 | |
| *** tzn has joined #openstack-infra | 22:14 | |
| *** zeih has quit IRC | 22:24 | |
| *** ashleighfarnham has joined #openstack-infra | 22:29 | |
| *** achanda has joined #openstack-infra | 22:47 | |
| *** markvoelker has joined #openstack-infra | 22:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:48 | |
| *** nwkarsten has joined #openstack-infra | 22:50 | |
| *** ashleighfarnham has quit IRC | 22:50 | |
| *** shashank_hegde has quit IRC | 22:50 | |
| *** shashank_hegde has joined #openstack-infra | 22:50 | |
| *** markvoelker has quit IRC | 22:52 | |
| *** nwkarsten has quit IRC | 22:54 | |
| *** thorst has joined #openstack-infra | 22:55 | |
| *** roxanaghe has joined #openstack-infra | 22:57 | |
| *** roxanagh_ has joined #openstack-infra | 22:58 | |
| *** thorst has quit IRC | 22:59 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 23:00 |
| *** thorst has joined #openstack-infra | 23:01 | |
| *** roxanaghe has quit IRC | 23:01 | |
| *** mriedem has joined #openstack-infra | 23:03 | |
| *** kzaitsev_mb has quit IRC | 23:04 | |
| *** roxanagh_ has quit IRC | 23:04 | |
| *** nelsnelson has joined #openstack-infra | 23:06 | |
| *** woodster_ has quit IRC | 23:06 | |
| *** thorst has quit IRC | 23:09 | |
| *** zz_dimtruck is now known as dimtruck | 23:10 | |
| *** dims has joined #openstack-infra | 23:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:23 | |
| *** e0ne has joined #openstack-infra | 23:23 | |
| *** ashleighfarnham has joined #openstack-infra | 23:27 | |
| *** kzaitsev_mb has quit IRC | 23:28 | |
| *** ashleighfarnham has quit IRC | 23:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:30 | |
| *** dims has quit IRC | 23:37 | |
| *** kzaitsev_mb has quit IRC | 23:47 | |
| *** markvoelker has joined #openstack-infra | 23:48 | |
| *** dimtruck is now known as zz_dimtruck | 23:50 | |
| *** e0ne has quit IRC | 23:54 | |
| *** dims has joined #openstack-infra | 23:55 | |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| *** yamamoto has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!