| clarkb | nodepool graceful shutdown can be slow... going to give it a bit longer before I take a bigger hammer to it | 00:00 |
|---|---|---|
| kiall | clarkb, no worries - openstack-infra downtime is the OpenStack equivalent of http://xkcd.com/303/ ;) | 00:01 |
| sdague | in going back through - code I pushed for review earlier today - https://review.openstack.org/#/c/84213/ | 00:02 |
| *** Hefeweiz1n has quit IRC | 00:02 | |
| *** andreaf has quit IRC | 00:02 | |
| sdague | looks like stable havana is broken on ceilometer vs. alembic | 00:03 |
| clarkb | sdague: wow it really loves us | 00:03 |
| *** gokrokve has quit IRC | 00:04 | |
| sdague | anyway, I need to call it a night | 00:06 |
| *** miguelzuniga has quit IRC | 00:08 | |
| *** mfer has joined #openstack-infra | 00:08 | |
| fungi | okay, back and reading scrollback | 00:10 |
| clarkb | fungi: tl;dr is /opt/nodepool was updated with broken code (proposed fix at https://review.openstack.org/#/c/84278/) so I have stopped puppet on nodepool, checked out ad7b9a849b1b2c7e6c9a7d28700e3952bfa18324 in/opt/nodepool, installed nodepoolm and applied lifeless' nodepool.yaml fix by hand and am waiting for graceful nodepool restart | 00:11 |
| *** dstanek has joined #openstack-infra | 00:14 | |
| lifeless | clarkb: how long does a graceful shutdown take? | 00:14 |
| clarkb | lifeless: it waits for all pending operations to complete | 00:14 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Report pecan changesets, merges, and +2s to #pecanpy https://review.openstack.org/83564 | 00:14 |
| *** khyati_ has joined #openstack-infra | 00:15 | |
| clarkb | lifeless: right now we are waiting on ~23 more nodes to finish building and whatever deletes are running to complete | 00:15 |
| clarkb | lifeless: I think we are about 5-10 minutes away at athis point | 00:15 |
| *** jswarren has quit IRC | 00:16 | |
| asselin | hi all, anyone have tips for nodepool to setup the http proxy configuration? via script? or predefine it in the image? | 00:17 |
| *** mgagne has joined #openstack-infra | 00:17 | |
| asselin | for the provider node | 00:17 |
| clarkb | asselin: simplest thing is probably to have the image build script set it for you | 00:20 |
| asselin | clarkb, yes, I did that and it works until it gets to here, when the env vars seem to get lost in the new bash session. wget https://git.openstack.org/cgit/openstack-infra/config/plain/install_puppet.sh | 00:21 |
| asselin | sudo bash -xe install_puppet.sh | 00:21 |
| clarkb | asselin: how are you setting the env vars? | 00:21 |
| asselin | export http_proxy="..." | 00:22 |
| asselin | just a few lines above in the script | 00:22 |
| asselin | and export https_proxy | 00:22 |
| clarkb | asselin: is that in the script that calls install_puppet.sh? | 00:23 |
| mordred | I have dealt with proxy in all parts of infra _except_ in nodepool | 00:23 |
| clarkb | down to 15 building nodes now | 00:23 |
| mordred | clarkb, asselin: sudo blowing away the environment is going to be the thing killing you there | 00:23 |
| clarkb | I guess my 10 minute guestimate was bad | 00:23 |
| clarkb | mordred: oh right | 00:23 |
| asselin | yes. this is in prepare_node.sh | 00:23 |
| fungi | clarkb: oh, so much fun... 84278 lgtm | 00:23 |
| mordred | you can fix it by adding -E to the sudo command - but then you're forking prepare_node.sh | 00:23 |
| clarkb | fungi: note, I haven't actually tested 84278 so I am just going back to the old stuff for now | 00:24 |
| asselin | mordred, yes...good point sudo might be clearing it out too | 00:24 |
| fungi | clarkb: "going back to the old stuff" means reverting all of the subnode-related patches, or what" | 00:24 |
| fungi | " | 00:24 |
| fungi | ? | 00:24 |
| clarkb | fungi: not reverting, I just checked out the code we were running on friday and stopped puppet | 00:25 |
| clarkb | and installed that code | 00:25 |
| *** ildikov_ has quit IRC | 00:25 | |
| fungi | clarkb: that will work with the migrated db and new-style config with labels (or are you not going back past the labels change)? | 00:25 |
| clarkb | I am not going past the labels change | 00:26 |
| *** gokrokve has joined #openstack-infra | 00:26 | |
| fungi | okay, fair enough | 00:26 |
| clarkb | fungi: ad7b9a849b1b2c7e6c9a7d28700e3952bfa18324 is the commit I am using | 00:26 |
| clarkb | down to 12 building | 00:26 |
| clarkb | 10 | 00:27 |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| clarkb | I expect it to be happy nyan cats after the graceful restart | 00:27 |
| fungi | having nodepool abort and delete building nodes on sigusr1 is probably in order to speed this up next | 00:28 |
| clarkb | ++ | 00:29 |
| clarkb | since they will all be deleted on start anyways | 00:29 |
| lifeless | why does nodepool delete everything on startup ? | 00:30 |
| *** qs201 has joined #openstack-infra | 00:31 | |
| *** bknudson has joined #openstack-infra | 00:31 | |
| clarkb | lifeless: it has to do with jenkins not cleaning them up otherwise | 00:31 |
| clarkb | iirc the issue is nodepool can miss jenkins events while it is off and thus must assume nodes hae been used | 00:31 |
| *** mfer has quit IRC | 00:32 | |
| clarkb | fungi: is there a better way to do the restart? | 00:32 |
| lifeless | clarkb: hmm | 00:32 |
| *** mfer has joined #openstack-infra | 00:33 | |
| lifeless | if only there was some way to find out whether a job has been run on a node | 00:33 |
| fungi | clarkb: i've ended up killing nodepoold via normal initscript stop (not graceful stop) and then deleting any building state nodes when it comes back up, and checking back to delete stale nodes a few hours later | 00:33 |
| clarkb | down to 5 building nodes now | 00:34 |
| clarkb | maybe we can let it do the restart on its own | 00:34 |
| clarkb | now 3 | 00:36 |
| clarkb | 2 | 00:37 |
| clarkb | its the final countdown! | 00:37 |
| lifeless | nah-na-naaah-nah | 00:37 |
| clarkb | 1! | 00:39 |
| *** JayF has quit IRC | 00:40 | |
| *** JayF has joined #openstack-infra | 00:41 | |
| clarkb | hrm building is down to 0, must be waiting on something else now | 00:41 |
| clarkb | a loop through the deletes? | 00:41 |
| clarkb | but at this point I Think it is relatively safe to kill with more fire | 00:41 |
| clarkb | fungi: ^ thoughts? | 00:41 |
| *** mfer has quit IRC | 00:43 | |
| lifeless | clarkb: since its going to delete it all anyhow... | 00:43 |
| lifeless | clarkb: there doesn't seem any benefit in waiting | 00:44 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add the correct pecan homepage for Github mirroring. https://review.openstack.org/84302 | 00:44 |
| clarkb | right, though I really expected it to stop at this point | 00:44 |
| clarkb | since it should have only been waiting on the nodes to launch to finish launching | 00:44 |
| *** markmcclain has quit IRC | 00:45 | |
| *** dstanek has quit IRC | 00:45 | |
| fungi | clarkb: moar fire | 00:46 |
| clarkb | thread dump indicates it is waiting on those deletions though | 00:46 |
| clarkb | fungi: ok proceeding with more fire | 00:47 |
| fungi | some of those deletes may never complete, i think | 00:47 |
| fungi | though not sure if it merely waits for them to time out | 00:47 |
| clarkb | ok it is doing stuff now | 00:48 |
| *** changbl has joined #openstack-infra | 00:48 | |
| clarkb | building 778 nodes... | 00:49 |
| *** amcrn has quit IRC | 00:49 | |
| *** CaptTofu has joined #openstack-infra | 00:49 | |
| mordred | wow | 00:52 |
| mordred | that's a lot of nodes | 00:52 |
| clarkb | woot we have new ready nodes finally | 00:53 |
| clarkb | and a bunch of nodes transitioning to used | 00:53 |
| *** markmcclain has joined #openstack-infra | 00:53 | |
| *** gothicmindfood has left #openstack-infra | 00:54 | |
| *** CaptTofu has quit IRC | 00:54 | |
| *** UtahDave has quit IRC | 00:54 | |
| clarkb | mordred: https://review.openstack.org/#/c/84278/ should fix this for us, but I haven't been able to test it | 00:54 |
| clarkb | was too busy making nodepool go again | 00:54 |
| mordred | clarkb: looking | 00:55 |
| mordred | clarkb: that patch looks completely reasonable | 00:55 |
| *** amcrn has joined #openstack-infra | 00:55 | |
| clarkb | nothing like finding a bunch of ENOENT errors after you restart nodepool | 00:56 |
| *** zhiyan_ is now known as zhiyan | 00:57 | |
| mordred | clarkb: +2 - I defer to you on thoughts of landing | 00:57 |
| clarkb | tripleo images are building | 00:58 |
| clarkb | apparently those got deleted? | 00:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Fixup tripleo nodepool config. https://review.openstack.org/84263 | 00:58 |
| *** Sukhdev has quit IRC | 00:58 | |
| clarkb | I am going to take this as an opportunity to walk home | 00:59 |
| * clarkb & | 00:59 | |
| *** zigo has quit IRC | 00:59 | |
| *** jepoy has joined #openstack-infra | 00:59 | |
| *** zigo has joined #openstack-infra | 01:01 | |
| lifeless | yay building images | 01:03 |
| *** alff_ has quit IRC | 01:06 | |
| *** alff has quit IRC | 01:06 | |
| *** mrodden has joined #openstack-infra | 01:13 | |
| asselin | mordred, clarkb thanks for the tips for the sudo proxy issues. I got past them. Next is pip proxy...will do that tomorrow. | 01:15 |
| *** ameade has joined #openstack-infra | 01:17 | |
| *** CaptTofu has joined #openstack-infra | 01:17 | |
| *** dstanek has joined #openstack-infra | 01:17 | |
| *** khyati_ has quit IRC | 01:17 | |
| *** thomasem has quit IRC | 01:20 | |
| *** ameade has quit IRC | 01:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:23 | |
| *** nosnos has joined #openstack-infra | 01:26 | |
| *** yaguang has joined #openstack-infra | 01:32 | |
| *** mbacchi has quit IRC | 01:34 | |
| tchaypo | clarkb: why weren't you at home already? | 01:39 |
| clarkb | I am one of those weirdos that sits in an office | 01:39 |
| tchaypo | If it's walking distance from home, that's probably not terrible | 01:40 |
| tchaypo | you get a comfy chair and the mindset-altering benefit of a dedicated "work" space, plus some exercise | 01:40 |
| clarkb | the exercise is quite important | 01:41 |
| persia | (short) commutes are good for thinking. If you aren't lucky enough to have one, faking it by taking a 20 minute walk a couple times a day tends to increase productivity. | 01:41 |
| tchaypo | I've been breaking my day into two stints - the morning at home for review/triage/updating existing changesets, then I head into the city for lunch and do the new work in the afternoon from a library | 01:41 |
| tchaypo | I find the break does me good, and the different environments and times-of-day work well for the different kinds of work | 01:42 |
| persia | Requires a nice library, but that sounds like a lovely model. | 01:43 |
| clarkb | as the weathwr gets nicer I intend on doing waterfront hacking | 01:44 |
| tchaypo | I've got a passel of libraries within range of me - two different universities, the city council has a few nice branches (which even have free wifi), and the state library (although I haven't been there yet) | 01:45 |
| tchaypo | If they get stale there's a few cow-orking spaces I'm looking at as well | 01:46 |
| clarkb | can sit in the park by mohai or on one of the many piers and tether | 01:46 |
| fungi | once i settle on a place with a dock, i plan to do *plenty* of bayside hacking | 01:47 |
| *** kiall has quit IRC | 01:47 | |
| fungi | well, technically it's a sound, not a bay, but close enough | 01:47 |
| fungi | calm enough for a dock regardless | 01:48 |
| persia | I used to go to a park by the river, but I found it hard to find a place that provided a good upright seat, a nice view, and didn't flood my laptop with sunlight. You may need to arrange strategically placed trees or a large umbrella, or something. | 01:49 |
| fungi | eh, i'll erect a sunshade or or something by the boat lift | 01:50 |
| *** CaptTofu has quit IRC | 01:51 | |
| *** _Alexandra_ has joined #openstack-infra | 01:54 | |
| *** jepoy has quit IRC | 01:54 | |
| lifeless | persia: o/ | 01:56 |
| *** CaptTofu has joined #openstack-infra | 01:58 | |
| persia | lifeless: Good afternoon. | 01:59 |
| *** wenlock has joined #openstack-infra | 02:01 | |
| *** dstanek has quit IRC | 02:01 | |
| *** zehicle_at_dell has quit IRC | 02:03 | |
| *** adalbas has quit IRC | 02:03 | |
| *** Ryan_Lane has quit IRC | 02:04 | |
| *** zhiyan has left #openstack-infra | 02:04 | |
| *** zhiyan has joined #openstack-infra | 02:05 | |
| tchaypo | lifeless merely hacks from his eyrie on a clifftop overlooking the peons below | 02:05 |
| tchaypo | or so I've heard | 02:05 |
| clarkb | I dunno skype made it look like a closet last we used video | 02:06 |
| *** weshay has quit IRC | 02:06 | |
| *** maxbit has joined #openstack-infra | 02:07 | |
| *** jepoy has joined #openstack-infra | 02:10 | |
| *** zehicle has joined #openstack-infra | 02:15 | |
| clarkb | jogo: large ops is broken | 02:18 |
| clarkb | connecting to blah blah blah via ssh timed out | 02:19 |
| clarkb | which is funny because those aren't real nodes | 02:19 |
| clarkb | did tempest somehow make a chagne to what the test lists are? | 02:19 |
| *** dims has quit IRC | 02:23 | |
| *** dims has joined #openstack-infra | 02:24 | |
| clarkb | jogo: looks like it is running smoke tests against the largeops job | 02:24 |
| clarkb | guessing d-g d61f684619763a42a849bb907934929a2cd711f6 is at faulkt | 02:25 |
| clarkb | huh maybe not that change passed large ops in the gate | 02:27 |
| *** dstanek has joined #openstack-infra | 02:27 | |
| clarkb | unless that is somehow not self gating | 02:28 |
| clarkb | http://logs.openstack.org/43/80643/5/gate/gate-tempest-dsvm-large-ops/ff376c2/logs/localrc.txt.gz sets it to 50 so I don't think that is the problem | 02:29 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add in train tracks to status page https://review.openstack.org/84314 | 02:30 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Ensure bootstrap's fonts are installed https://review.openstack.org/84315 | 02:30 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Tweak job layout css in status page https://review.openstack.org/84316 | 02:30 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Adjust change headings on status page https://review.openstack.org/84317 | 02:30 |
| *** amotoki_ has quit IRC | 02:30 | |
| clarkb | I see the problem, patch incoming | 02:31 |
| lifeless | tchaypo: if by peons you mean my cats | 02:34 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Enable large ops if flag is >= 1. https://review.openstack.org/84319 | 02:34 |
| clarkb | jogo: fungi jeblair mordred SergeyLukjanov ^ | 02:34 |
| clarkb | I will probably self approve that once the largeops test runs and confirms that fixes the problem | 02:34 |
| StevenK | clarkb: "Maybe the comparison flag -ge 1 to catch non default cases." ? :-) | 02:35 |
| clarkb | ugh did I tyop it? | 02:36 |
| StevenK | clarkb: Yeah, in the commit message | 02:36 |
| StevenK | But y and b are no where near k :-P | 02:36 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Enable large ops if flag is >= 1. https://review.openstack.org/84319 | 02:37 |
| clarkb | StevenK: there ^ | 02:37 |
| clarkb | StevenK: I have a faulty typing driver | 02:37 |
| clarkb | it caches movements and sometimes the cache eviction algorithm fails and it spits out random crap | 02:37 |
| *** timrc is now known as timrc-afk | 02:38 | |
| lifeless | clarkb: / fungi: is nodepool happy w/tripleo cloud ? | 02:41 |
| *** dstanek has quit IRC | 02:42 | |
| lifeless | I see some error VMs being deleted and 3 new ones spawning | 02:42 |
| *** amcrn has quit IRC | 02:42 | |
| *** david-lyle has joined #openstack-infra | 02:44 | |
| clarkb | lifeless: I don't currently have access to those logs, looks like 3 nodes started tests in the last ~5 minutes though | 02:45 |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add --force to transifex pull https://review.openstack.org/84323 | 02:56 |
| clarkb | I have self approved the d-g change that fixes large ops, it just passed its own large ops test run | 02:57 |
| *** markmcclain has quit IRC | 02:59 | |
| *** harlowja is now known as harlowja_away | 02:59 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add a slave-script to Jenkins for pushing to swift https://review.openstack.org/71189 | 03:01 |
| *** lcheng has joined #openstack-infra | 03:06 | |
| nibalizer | i just changed my launchpad username from krum-spencer to nibalizer, will I have any trouble with openstack services? | 03:08 |
| clarkb | nibalizer: you might, depends on whether or not launchpad gievs you a new openid endpoint | 03:09 |
| clarkb | I never remember what circumstances that happens under | 03:09 |
| nibalizer | i was able to review | 03:09 |
| *** harlowja_away is now known as harlowja | 03:09 | |
| nibalizer | so hopefully we're good | 03:09 |
| clarkb | nibalizer: the CLA stuff is usually what breaks | 03:09 |
| *** khyati_ has joined #openstack-infra | 03:10 | |
| *** chandan_kumar has joined #openstack-infra | 03:11 | |
| nibalizer | hrm i'll try and push a change | 03:12 |
| nibalizer | ya thats what i'm worried about | 03:12 |
| *** yamahata has joined #openstack-infra | 03:13 | |
| clarkb | dstufft: are you around? http://logs.openstack.org/78/84278/1/check/gate-nodepool-docs/3acec62/console.html has me confused, I can reproduce that using tox, but not pip 1.5 directly | 03:14 |
| clarkb | dstufft: basically test-requirements says sphinx>=1.1.2,<1.2 and 1.2b3 is being installed which breaks us | 03:14 |
| clarkb | dstufft: with my manual testing I have installed every dep individually looking for the ones that install sphinx and it doesn't appear to install sphinx until it is explicitly called out in test-requirements.txt so I don't think this is transitive weirdness | 03:15 |
| clarkb | dstufft: any idea why Sphinx==1.2b3 is being installed? | 03:15 |
| *** timrc-afk is now known as timrc | 03:16 | |
| tchaypo | lifeless: no, cats would be overlords | 03:17 |
| lifeless | clarkb: 1.2b3 is < 1.2 | 03:18 |
| clarkb | lifeless: yes, but pip shouldn't install beta releases without explicitly being told to | 03:18 |
| clarkb | oh I know | 03:18 |
| clarkb | tox tells pip to do that | 03:18 |
| lifeless | clarkb: as long as its properly detecting they are betas, sure. | 03:18 |
| clarkb | because tox | 03:18 |
| lifeless | clarkb: or that ;) | 03:19 |
| *** matsuhashi has quit IRC | 03:22 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Fix tox's insane pip install command. https://review.openstack.org/84327 | 03:23 |
| *** timrc is now known as timrc-afk | 03:28 | |
| *** mgagne has quit IRC | 03:32 | |
| *** Ryan_Lane has joined #openstack-infra | 03:33 | |
| *** chandan_kumar has quit IRC | 03:35 | |
| *** nosnos has quit IRC | 03:37 | |
| *** alff_ has joined #openstack-infra | 03:38 | |
| *** alff has joined #openstack-infra | 03:38 | |
| *** alexpilotti has joined #openstack-infra | 03:40 | |
| *** CaptTofu has quit IRC | 03:41 | |
| *** markwash has quit IRC | 03:47 | |
| *** markwash has joined #openstack-infra | 03:50 | |
| *** alexpilotti has quit IRC | 03:52 | |
| *** _Alexandra_ has quit IRC | 03:56 | |
| *** chandan_kumar has joined #openstack-infra | 03:59 | |
| *** chandan_kumar has quit IRC | 04:01 | |
| *** lcheng has quit IRC | 04:03 | |
| *** chandan_kumar has joined #openstack-infra | 04:06 | |
| *** _Alexandra_ has joined #openstack-infra | 04:07 | |
| clarkb | lifeless: so I just realized, when I applied your change to nodepool.yaml I did it by hand and may not have bumped the max servers which would explain the behavior we are seeing | 04:09 |
| clarkb | lifeless: I can fix that tomorrow | 04:09 |
| *** _Alexandra_ has quit IRC | 04:12 | |
| *** thuc has joined #openstack-infra | 04:13 | |
| *** thuc_ has joined #openstack-infra | 04:13 | |
| *** matsuhashi has joined #openstack-infra | 04:14 | |
| *** thuc has quit IRC | 04:17 | |
| *** thuc has joined #openstack-infra | 04:21 | |
| *** nosnos has joined #openstack-infra | 04:23 | |
| *** thuc_ has quit IRC | 04:24 | |
| *** dstanek has joined #openstack-infra | 04:28 | |
| *** dstanek has quit IRC | 04:33 | |
| *** ArxCruz has quit IRC | 04:33 | |
| *** ArxCruz has joined #openstack-infra | 04:35 | |
| *** sandywalsh_ has quit IRC | 04:44 | |
| *** sandywalsh has joined #openstack-infra | 04:46 | |
| *** gokrokve_ has joined #openstack-infra | 04:50 | |
| *** alff has quit IRC | 04:52 | |
| *** alff_ has quit IRC | 04:52 | |
| *** gokrokve has quit IRC | 04:53 | |
| *** mihgen has joined #openstack-infra | 04:55 | |
| *** zul has quit IRC | 05:00 | |
| lifeless | clarkb: ah yes that would explain it | 05:00 |
| *** cody-somerville has quit IRC | 05:01 | |
| *** matsuhas_ has joined #openstack-infra | 05:02 | |
| *** matsuhas_ has quit IRC | 05:02 | |
| *** cody-somerville has joined #openstack-infra | 05:02 | |
| *** mrmartin has joined #openstack-infra | 05:02 | |
| *** zul has joined #openstack-infra | 05:04 | |
| *** matsuhashi has quit IRC | 05:05 | |
| *** matsuhashi has joined #openstack-infra | 05:05 | |
| *** harlowja is now known as harlowja_away | 05:11 | |
| *** thuc has quit IRC | 05:13 | |
| *** thuc has joined #openstack-infra | 05:14 | |
| *** thuc has quit IRC | 05:18 | |
| *** ildikov_ has joined #openstack-infra | 05:20 | |
| *** gyee has quit IRC | 05:20 | |
| *** qs201 has quit IRC | 05:21 | |
| *** khyati_ has quit IRC | 05:22 | |
| *** mrmartin has quit IRC | 05:27 | |
| *** CaptTofu has joined #openstack-infra | 05:30 | |
| *** CaptTofu has quit IRC | 05:35 | |
| *** zhiyan is now known as zhiyan_ | 05:42 | |
| *** peddamat has joined #openstack-infra | 05:46 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Nothing to see here. Move along https://review.openstack.org/84356 | 05:47 |
| *** _nadya_ has joined #openstack-infra | 05:48 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Enable large ops if flag is >= 1 https://review.openstack.org/84319 | 05:54 |
| clarkb | for anyone that wakes up before me in the morning, ^ will fix the devstack-gate trouble. I pushed a new patchset and re self approved because it failed in the gate and this was the quickest way to turn it around and get it back into the gate without simply merging it | 05:55 |
| *** pcrews_ has quit IRC | 06:00 | |
| *** Ryan_Lane1 has joined #openstack-infra | 06:04 | |
| *** _nadya_ has quit IRC | 06:07 | |
| *** basha has joined #openstack-infra | 06:08 | |
| *** saju_m has joined #openstack-infra | 06:21 | |
| *** e0ne has joined #openstack-infra | 06:23 | |
| *** thuc has joined #openstack-infra | 06:24 | |
| *** dstanek has joined #openstack-infra | 06:27 | |
| *** _nadya_ has joined #openstack-infra | 06:27 | |
| *** thuc has quit IRC | 06:29 | |
| *** imcsk8_lap has quit IRC | 06:30 | |
| *** imcsk8_lap has joined #openstack-infra | 06:30 | |
| *** _nadya_ has quit IRC | 06:31 | |
| *** dstanek has quit IRC | 06:31 | |
| *** _nadya_ has joined #openstack-infra | 06:33 | |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Remove experimental trove devstack-gate job https://review.openstack.org/84369 | 06:36 |
| *** e0ne has quit IRC | 06:37 | |
| *** AJaeger has quit IRC | 06:38 | |
| *** nosnos has quit IRC | 06:45 | |
| *** nosnos has joined #openstack-infra | 06:46 | |
| *** mihgen has quit IRC | 06:48 | |
| *** flaper87|afk is now known as flaper87 | 06:50 | |
| *** mkoderer has joined #openstack-infra | 06:51 | |
| *** jhesketh has quit IRC | 06:55 | |
| *** matsuhashi has quit IRC | 06:58 | |
| *** matsuhashi has joined #openstack-infra | 06:58 | |
| *** _nadya_ has joined #openstack-infra | 07:01 | |
| *** matsuhashi has quit IRC | 07:03 | |
| *** _nadya_ has quit IRC | 07:03 | |
| *** Ryan_Lane1 has quit IRC | 07:04 | |
| *** yolanda has joined #openstack-infra | 07:05 | |
| *** mrmartin has joined #openstack-infra | 07:05 | |
| *** matsuhashi has joined #openstack-infra | 07:06 | |
| *** Ryan_Lane has quit IRC | 07:07 | |
| *** jhesketh has joined #openstack-infra | 07:08 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add new job to build misc Japanese Guides https://review.openstack.org/83979 | 07:09 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Remove py3 gate job from kite https://review.openstack.org/84372 | 07:13 |
| *** jcoufal has joined #openstack-infra | 07:16 | |
| *** CaptTofu has joined #openstack-infra | 07:19 | |
| *** sandywalsh_ has joined #openstack-infra | 07:22 | |
| *** CaptTofu has quit IRC | 07:23 | |
| *** sandywalsh has quit IRC | 07:25 | |
| *** jamielennox is now known as jamielennox|away | 07:27 | |
| *** mrda is now known as mrda_away | 07:29 | |
| *** basha has quit IRC | 07:29 | |
| *** salv-orlando has quit IRC | 07:30 | |
| *** salv-orlando has joined #openstack-infra | 07:30 | |
| *** alff_ has joined #openstack-infra | 07:32 | |
| *** alff has joined #openstack-infra | 07:32 | |
| *** mihgen has joined #openstack-infra | 07:35 | |
| *** marun is now known as maru_afk | 07:36 | |
| *** jgallard has joined #openstack-infra | 07:37 | |
| *** gokrokve_ has quit IRC | 07:40 | |
| *** gokrokve has joined #openstack-infra | 07:41 | |
| *** hashar has joined #openstack-infra | 07:44 | |
| *** gokrokve has quit IRC | 07:45 | |
| *** basha has joined #openstack-infra | 07:47 | |
| *** jpich has joined #openstack-infra | 08:02 | |
| *** matsuhashi has quit IRC | 08:03 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Enable large ops if flag is >= 1 https://review.openstack.org/84319 | 08:04 |
| *** matsuhas_ has joined #openstack-infra | 08:05 | |
| *** dizquierdo has joined #openstack-infra | 08:14 | |
| *** liam__ has joined #openstack-infra | 08:18 | |
| *** liam__ has quit IRC | 08:19 | |
| *** CaptTofu has joined #openstack-infra | 08:20 | |
| *** derekh has joined #openstack-infra | 08:24 | |
| *** CaptTofu has quit IRC | 08:24 | |
| *** matsuhas_ has quit IRC | 08:33 | |
| *** jlibosva has joined #openstack-infra | 08:35 | |
| *** andreaf has joined #openstack-infra | 08:37 | |
| *** zhiyan_ is now known as zhiyan | 08:38 | |
| *** matsuhashi has joined #openstack-infra | 08:38 | |
| *** alff_ has quit IRC | 08:40 | |
| *** alff has quit IRC | 08:40 | |
| *** saju_m has quit IRC | 08:43 | |
| *** peddamat has quit IRC | 08:45 | |
| *** jp_at_hp has joined #openstack-infra | 08:53 | |
| *** jpich has quit IRC | 08:55 | |
| *** saju_m has joined #openstack-infra | 08:56 | |
| *** afazekas has joined #openstack-infra | 08:57 | |
| *** yonglihe_ has quit IRC | 09:00 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/devstack-gate: Only install libvirt if the driver is libvirt(!) https://review.openstack.org/84381 | 09:01 |
| *** jpich has joined #openstack-infra | 09:07 | |
| *** jooools has joined #openstack-infra | 09:10 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/devstack-gate: Only install libvirt if the driver is libvirt(!) https://review.openstack.org/84381 | 09:14 |
| *** gokrokve has joined #openstack-infra | 09:16 | |
| *** jcoufal has quit IRC | 09:18 | |
| *** jooools has quit IRC | 09:20 | |
| *** gokrokve has quit IRC | 09:21 | |
| *** jooools has joined #openstack-infra | 09:22 | |
| *** chandankumar_ has joined #openstack-infra | 09:25 | |
| *** dstanek has joined #openstack-infra | 09:30 | |
| *** dstanek has quit IRC | 09:34 | |
| *** dizquierdo is now known as dizquierdo_afk | 09:42 | |
| *** morganfainberg is now known as morganfainberg_Z | 09:46 | |
| *** gokrokve has joined #openstack-infra | 09:47 | |
| *** jcoufal has joined #openstack-infra | 09:50 | |
| *** gokrokve has quit IRC | 09:52 | |
| *** chandankumar_ has quit IRC | 09:54 | |
| *** david-lyle has quit IRC | 09:55 | |
| *** jepoy has quit IRC | 09:56 | |
| *** jepoy has joined #openstack-infra | 09:56 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Enable sudo for rally-install job https://review.openstack.org/84394 | 09:58 |
| *** ihrachys is now known as ihrachys|lunch | 09:59 | |
| *** jepoy has quit IRC | 10:01 | |
| *** e0ne has joined #openstack-infra | 10:03 | |
| *** CaptTofu has joined #openstack-infra | 10:09 | |
| *** mrmartin has quit IRC | 10:11 | |
| *** e0ne has quit IRC | 10:13 | |
| *** wenlock has quit IRC | 10:13 | |
| *** CaptTofu has quit IRC | 10:13 | |
| *** matsuhashi has quit IRC | 10:14 | |
| *** jgallard has quit IRC | 10:20 | |
| *** yamahata has quit IRC | 10:24 | |
| *** e0ne has joined #openstack-infra | 10:24 | |
| *** e0ne_ has joined #openstack-infra | 10:30 | |
| *** DuncanT- has quit IRC | 10:31 | |
| *** e0ne has quit IRC | 10:33 | |
| *** hashar has quit IRC | 10:33 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: normalize the services for a post grizzly world https://review.openstack.org/84287 | 10:35 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: turn back on log enforcement https://review.openstack.org/81216 | 10:35 |
| *** nosnos has quit IRC | 10:40 | |
| *** yaguang has quit IRC | 10:43 | |
| *** markmc has joined #openstack-infra | 10:45 | |
| *** CaptTofu has joined #openstack-infra | 10:46 | |
| *** pblaho has joined #openstack-infra | 10:48 | |
| *** gokrokve has joined #openstack-infra | 10:48 | |
| *** dizquierdo_afk is now known as dizquierdo | 10:49 | |
| *** mrmartin_ has joined #openstack-infra | 10:52 | |
| *** gokrokve has quit IRC | 10:53 | |
| *** CaptTofu has quit IRC | 10:54 | |
| *** yaguang has joined #openstack-infra | 10:55 | |
| *** roeyc has joined #openstack-infra | 11:06 | |
| amotoki | hi, i noticed OpenStack Style Guidelines still has vim modelines http://docs.openstack.org/developer/hacking/ | 11:08 |
| amotoki | though it is removed in the latest hacking repo but it is not released yet. | 11:08 |
| amotoki | it seems it is better to be updated. | 11:08 |
| *** mbacchi has joined #openstack-infra | 11:10 | |
| *** bookwar has joined #openstack-infra | 11:16 | |
| *** sandywalsh_ is now known as sandywalsh | 11:18 | |
| *** e0ne_ has quit IRC | 11:22 | |
| *** ihrachys|lunch is now known as ihrachys | 11:24 | |
| *** jgallard has joined #openstack-infra | 11:24 | |
| *** hashar has joined #openstack-infra | 11:28 | |
| *** maxbit has quit IRC | 11:30 | |
| *** ominakov has joined #openstack-infra | 11:30 | |
| *** ociuhandu has joined #openstack-infra | 11:33 | |
| *** alexpilotti has joined #openstack-infra | 11:34 | |
| *** kiall_ is now known as Kiall | 11:36 | |
| *** ams0 has joined #openstack-infra | 11:40 | |
| *** moted has quit IRC | 11:40 | |
| *** weshay has joined #openstack-infra | 11:41 | |
| *** ameade has joined #openstack-infra | 11:42 | |
| *** yolanda has quit IRC | 11:42 | |
| *** moted has joined #openstack-infra | 11:44 | |
| *** yaguang has quit IRC | 11:44 | |
| *** saju_m has quit IRC | 11:44 | |
| *** maxbit has joined #openstack-infra | 11:44 | |
| *** ams0 has quit IRC | 11:46 | |
| openstackgerrit | A change was merged to openstack/requirements: Using happybase >=0.5,!=0.7 https://review.openstack.org/82438 | 11:47 |
| *** gokrokve has joined #openstack-infra | 11:47 | |
| *** gokrokve has quit IRC | 11:52 | |
| *** jgallard has quit IRC | 11:52 | |
| *** jgallard has joined #openstack-infra | 11:52 | |
| *** yolanda has joined #openstack-infra | 11:54 | |
| *** saju_m has joined #openstack-infra | 11:56 | |
| *** mflobo has joined #openstack-infra | 11:57 | |
| *** maxbit has quit IRC | 12:00 | |
| *** pdmars has joined #openstack-infra | 12:01 | |
| *** saju_m has quit IRC | 12:02 | |
| *** e0ne has joined #openstack-infra | 12:02 | |
| *** peddamat has joined #openstack-infra | 12:02 | |
| *** peddamat has quit IRC | 12:02 | |
| *** rfolco has joined #openstack-infra | 12:03 | |
| *** rfolco has quit IRC | 12:03 | |
| *** e0ne has quit IRC | 12:04 | |
| *** e0ne has joined #openstack-infra | 12:04 | |
| *** afazekas has quit IRC | 12:04 | |
| *** rfolco has joined #openstack-infra | 12:04 | |
| *** che-arne has joined #openstack-infra | 12:08 | |
| *** CaptTofu has joined #openstack-infra | 12:08 | |
| *** dkranz has joined #openstack-infra | 12:09 | |
| *** e0ne has quit IRC | 12:09 | |
| *** amotoki has quit IRC | 12:11 | |
| *** CaptTofu has quit IRC | 12:16 | |
| *** jepoy has joined #openstack-infra | 12:16 | |
| *** jhesketh has quit IRC | 12:16 | |
| *** eharney has joined #openstack-infra | 12:16 | |
| *** dcramer_ has quit IRC | 12:17 | |
| *** afazekas has joined #openstack-infra | 12:18 | |
| *** e0ne has joined #openstack-infra | 12:19 | |
| *** saju_m has joined #openstack-infra | 12:21 | |
| *** dims has quit IRC | 12:22 | |
| *** saju_m has quit IRC | 12:23 | |
| *** thomasem has joined #openstack-infra | 12:26 | |
| *** thomasem_ has joined #openstack-infra | 12:27 | |
| *** markmcclain has joined #openstack-infra | 12:28 | |
| *** maxbit has joined #openstack-infra | 12:29 | |
| *** aysyd has joined #openstack-infra | 12:30 | |
| *** thomasem has quit IRC | 12:31 | |
| *** dstanek has joined #openstack-infra | 12:32 | |
| fungi | and so it begins | 12:33 |
| *** basha has quit IRC | 12:33 | |
| fifieldt | duh duh duuuuuuuhhh | 12:34 |
| *** dims_ has joined #openstack-infra | 12:34 | |
| *** rlandy has joined #openstack-infra | 12:34 | |
| *** dstanek has quit IRC | 12:36 | |
| *** CaptTofu has joined #openstack-infra | 12:37 | |
| *** timrc-afk is now known as timrc | 12:38 | |
| *** saju_m has joined #openstack-infra | 12:39 | |
| *** jgrimm has joined #openstack-infra | 12:43 | |
| *** signed8bit has joined #openstack-infra | 12:47 | |
| *** gokrokve has joined #openstack-infra | 12:47 | |
| *** signed8bit has quit IRC | 12:48 | |
| *** thuc has joined #openstack-infra | 12:50 | |
| *** dkliban has quit IRC | 12:51 | |
| *** gokrokve has quit IRC | 12:52 | |
| *** thuc_ has joined #openstack-infra | 12:52 | |
| *** thuc has quit IRC | 12:55 | |
| *** dprince has joined #openstack-infra | 12:57 | |
| *** heyongli has joined #openstack-infra | 13:01 | |
| *** signed8bit has joined #openstack-infra | 13:03 | |
| *** mrmartin_ has quit IRC | 13:03 | |
| *** zehicle has quit IRC | 13:07 | |
| *** dstanek has joined #openstack-infra | 13:07 | |
| *** pblaho has quit IRC | 13:08 | |
| *** dkranz has quit IRC | 13:10 | |
| *** jeckersb_gone is now known as jeckersb | 13:11 | |
| *** afazekas has quit IRC | 13:13 | |
| *** saju_m has quit IRC | 13:15 | |
| *** mriedem has joined #openstack-infra | 13:15 | |
| *** mflobo has quit IRC | 13:16 | |
| *** dcramer_ has joined #openstack-infra | 13:18 | |
| *** signed8bit has quit IRC | 13:18 | |
| *** eharney has quit IRC | 13:21 | |
| *** CaptTofu has quit IRC | 13:23 | |
| *** dkranz has joined #openstack-infra | 13:24 | |
| *** saju_m has joined #openstack-infra | 13:28 | |
| *** esker has joined #openstack-infra | 13:31 | |
| *** dkliban has joined #openstack-infra | 13:31 | |
| *** signed8bit has joined #openstack-infra | 13:34 | |
| *** zns has joined #openstack-infra | 13:34 | |
| *** jgrimm has quit IRC | 13:34 | |
| *** changbl has quit IRC | 13:35 | |
| *** julim has joined #openstack-infra | 13:38 | |
| *** mwagner_zzz has quit IRC | 13:40 | |
| *** nkinder has quit IRC | 13:40 | |
| *** thuc_ has quit IRC | 13:40 | |
| *** thuc has joined #openstack-infra | 13:41 | |
| *** prad has joined #openstack-infra | 13:41 | |
| *** thuc has quit IRC | 13:45 | |
| *** markmcclain has quit IRC | 13:45 | |
| *** thuc has joined #openstack-infra | 13:45 | |
| *** vhoward- has left #openstack-infra | 13:46 | |
| *** signed8bit has quit IRC | 13:46 | |
| *** afazekas has joined #openstack-infra | 13:46 | |
| *** vhoward- has joined #openstack-infra | 13:46 | |
| *** signed8bit has joined #openstack-infra | 13:47 | |
| *** gokrokve has joined #openstack-infra | 13:47 | |
| *** lcostantino has joined #openstack-infra | 13:49 | |
| *** gokrokve_ has joined #openstack-infra | 13:49 | |
| *** pcrews_ has joined #openstack-infra | 13:49 | |
| *** ams0 has joined #openstack-infra | 13:50 | |
| *** gokrokve has quit IRC | 13:52 | |
| *** gokrokve_ has quit IRC | 13:54 | |
| *** ams0 has quit IRC | 13:56 | |
| *** afazekas has quit IRC | 13:57 | |
| *** zns has quit IRC | 14:01 | |
| *** heyongli has quit IRC | 14:05 | |
| sdague | fungi: is the ES cluster ok? | 14:07 |
| sdague | I'm getting more proxy timeouts than I expect | 14:07 |
| fungi | sdague: i'll check | 14:07 |
| *** yassine has joined #openstack-infra | 14:09 | |
| *** zns has joined #openstack-infra | 14:11 | |
| fungi | sdague: i don't see any obvious signs of trouble... cluster health is green, no missing members, no missing workers, no shards being shuffled or anything | 14:11 |
| sdague | hmmm | 14:11 |
| sdague | ok | 14:11 |
| fungi | we're not backed up on tasks in the queue | 14:11 |
| sdague | yep | 14:11 |
| sdague | was there a kibana update recently? | 14:12 |
| Kiall | fungi: could I trouble you for a quick review on a gerrit ACL change? https://review.openstack.org/#/c/84273/ :) | 14:12 |
| sdague | the moment I try to search the 24hr bucket, I basically time out | 14:13 |
| sdague | and the browser gets killed with whatever gets handed back | 14:13 |
| *** saju_m has quit IRC | 14:14 | |
| fungi | sdague: ohh... i saw someone mention in here the other day that something might be wrong with multi-line handling and that some log files were getting treated as one line or something like that | 14:15 |
| fungi | yesterday maybe | 14:15 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova ec2 bug 1280532 https://review.openstack.org/84449 | 14:15 |
| uvirtbot | Launchpad bug 1280532 in nova "Detach volume fails with "Unexpected KeyError" in EC2 interface." [High,In progress] https://launchpad.net/bugs/1280532 | 14:15 |
| *** rcleere has joined #openstack-infra | 14:16 | |
| fungi | perhaps a change in some log format confused the multi-line filter, or could we have recently changed the filter? | 14:16 |
| sdague | oh, right | 14:16 |
| *** roeyc has quit IRC | 14:18 | |
| *** alff_ has joined #openstack-infra | 14:18 | |
| *** alff has joined #openstack-infra | 14:18 | |
| *** basha has joined #openstack-infra | 14:19 | |
| fungi | Kiall: are you actually expecting to have milestone-proposed branches in python-designateclient? none of the official openstack python-fooclient projects do that as far as i know | 14:19 |
| *** jgrimm has joined #openstack-infra | 14:20 | |
| fungi | Kiall: and milestone-proposed makes very little sense on a project unless you're going to use it as a staging area for future stable branches (which aren't reflected in the python-designateclient acl) | 14:20 |
| *** markmcclain has joined #openstack-infra | 14:21 | |
| *** adalbas has joined #openstack-infra | 14:21 | |
| Kiall | fungi: your're probably right about the client, but that side was just keeping things synced up | 14:21 |
| Kiall | (e.g. All-Projects does that for everything, and just overriding the Release Managers group here for designate-milesone) | 14:22 |
| *** eharney has joined #openstack-infra | 14:23 | |
| fungi | Kiall: well, all-projects does a lot of things which only apply if you have a particular branch on your project. you chose not to do a stable/* ref section (which is also something all-projects has) so why milestone-proposed? | 14:23 |
| *** basha has quit IRC | 14:23 | |
| Kiall | Heh - I missed that obviously -_- Either way, I think your right about just removing that section. | 14:24 |
| fungi | Kiall: if you want to respin it without that section now, i'll go ahead and approve it for you | 14:24 |
| sdague | fungi: so, how do you feel about a ninja merge of this then? - https://review.openstack.org/#/c/84213/ | 14:24 |
| Kiall | already on it, thanks :) | 14:24 |
| sdague | because that's the fix for the kibana issue | 14:25 |
| sdague | however it's deadlocked on ceilometer breaking stable | 14:25 |
| fungi | Kiall: it makes perfect sense on designate since that's a server project so you presumably expect to follow a release workflow similar to the other integrated servers, just not really relevant on python-designateclient | 14:25 |
| fungi | sdague: looking, but probably yes | 14:25 |
| fungi | sdague: on a related note, do you have any details on what ceilo needs to be doing to fix their migrations in havana? | 14:26 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Sync up designate repo ACLs https://review.openstack.org/84273 | 14:27 |
| *** shardy is now known as shardy_afk | 14:28 | |
| sdague | fungi: so actually jd__ has a fix in the gate now | 14:28 |
| anteaya | Kiall: how many people in the designate-milestone group? | 14:28 |
| Kiall | Okay .. that should be the minimal set of items in the 2x designate ACLs without needlessly overriding anything we're already inheriting from All-Projects | 14:29 |
| sdague | alembic now requires code in sqla >= 9 | 14:29 |
| sdague | so it's a requirements fix | 14:29 |
| anteaya | just wondering why designate needs more that -ptl and -core | 14:29 |
| sdague | which actually means we should be able to fix this the real way | 14:29 |
| Kiall | anteaya: right now it includes all the designate-core people, but may change over time | 14:29 |
| *** skraynev is now known as skraynev_afk | 14:30 | |
| fungi | anteaya: looks like if you merge it with the -core group (which is listed as an included group) and then de-dupe the results it's four humans | 14:30 |
| fungi | so not a ton | 14:30 |
| *** nkinder has joined #openstack-infra | 14:30 | |
| Kiall | Also - the group is pre-exiting | 14:30 |
| fungi | Kiall: but anteaya does have a point, that you may want to consider the -milestone group roughly equivalent to the ptl and list of delegates performing release management duties, and drop the -core inclusion on it | 14:30 |
| Kiall | Yea, it just so happens that today those are the same group of people | 14:31 |
| fungi | though that's something you can do entirely through gerrit's group management interface | 14:31 |
| fungi | Kiall: note that changes to acls aren't going to take effect until this friday when we run manage-projects manually | 14:32 |
| Kiall | fungi: ah.. it's disabled at the moment? | 14:32 |
| anteaya | I'm comfy with an individual doing tagging and other release management things, not really a group | 14:33 |
| anteaya | an individual being the -ptl | 14:33 |
| fungi | Kiall: yes, mordred and others are working on fixing some remaining race conditions on new project creation, so we don't run manage-projects automatically at the moment | 14:33 |
| Kiall | In the meantime, could I ask you to cut a branch on stackforge/designate using 2445db42520f828cd19255a68538fa094c6cad2b - at least reviews can pile up there until Friday | 14:33 |
| fungi | Kiall: gladly | 14:33 |
| Kiall | excellent :) | 14:34 |
| fungi | sdague: https://review.openstack.org/84433 i guess | 14:34 |
| *** david-lyle has joined #openstack-infra | 14:35 | |
| *** basha has joined #openstack-infra | 14:35 | |
| fungi | sdague: i can enqueue 84213 into the gate bypassing the check pipeline requirement if it's urgent to get it in before 84433 merges | 14:35 |
| *** CaptTofu has joined #openstack-infra | 14:36 | |
| fungi | which should be in ~45 minutes if we're lucky | 14:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Sync up designate repo ACLs https://review.openstack.org/84273 | 14:36 |
| sdague | fungi: no, it can merge after | 14:36 |
| *** afazekas has joined #openstack-infra | 14:37 | |
| fungi | Kiall: you want 2445db4 as the base of milestone-proposed or some other branch name? | 14:37 |
| sdague | actually, given that it was a requirements fix, this should probably work now. Let me redo results | 14:38 |
| Kiall | fungi: Yes, 2445db4 as the base of milestone-proposed is right | 14:38 |
| fungi | Kiall: done | 14:38 |
| Kiall | Thanks :) | 14:39 |
| fungi | Kiall: you're welcome, and your patience is appreciated as always! | 14:39 |
| *** CaptTofu has quit IRC | 14:40 | |
| fungi | as luck would have it, when there's one zone/region where we're having trouble getting image updates to complete, the job i want to re-run always seems to end up grabbed by a node from that one stale provider | 14:41 |
| anteaya | does that zone/region have a name? | 14:41 |
| anteaya | is it az-2 still? | 14:41 |
| fungi | anteaya: it's your old friend hpcloud-az2 | 14:41 |
| fungi | yes | 14:41 |
| anteaya | k | 14:42 |
| *** thuc_ has joined #openstack-infra | 14:42 | |
| fungi | there were some other rax and hpcloud zones where the overnight image updates haven't succeeded for a day or two, but i was able to get images built in those after a few manual retries | 14:42 |
| anteaya | go you | 14:42 |
| anteaya | but az-2 still stonewalls | 14:42 |
| anteaya | sigh | 14:42 |
| fungi | i'm just going to start a loop there retrying to get a new image until it sticks | 14:43 |
| fungi | last time this happened, it had to cycle continuously for about 36 hours to get one to complete | 14:43 |
| Kiall | fungi: any idea why it's failing? | 14:45 |
| anteaya | wow | 14:45 |
| anteaya | like testr --until-failure | 14:45 |
| fungi | Kiall: timeouts from facter trying to query the metadata server in that az | 14:45 |
| anteaya | only az-2 --until-image | 14:45 |
| *** thuc has quit IRC | 14:45 | |
| *** wenlock has joined #openstack-infra | 14:45 | |
| Kiall | fungi: ah yes, my good old friend the metadata service -_- | 14:46 |
| *** thuc_ has quit IRC | 14:46 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova ec2 bug 1280532 https://review.openstack.org/84449 | 14:47 |
| *** gokrokve has joined #openstack-infra | 14:47 | |
| uvirtbot | Launchpad bug 1280532 in nova "Detach volume fails with "Unexpected KeyError" in EC2 interface." [High,In progress] https://launchpad.net/bugs/1280532 | 14:47 |
| Kiall | I'm actually surprised nodepool does it that way, rather than using DIB and uploading the same image everywhere | 14:47 |
| fungi | Kiall: we haven't had glance available in our providers | 14:47 |
| fungi | Kiall: so we had to design it launching a template instance off a base image in the provider's existing catalog, modify that, shut it down and snapshot it | 14:48 |
| Kiall | Oh.. Well, yea - HP Cloud's new regions give you glance access if you ask for it | 14:49 |
| Kiall | No clue if RAX does the same though | 14:50 |
| fungi | Kiall: yep, and we have (and have tested it now successfully), but the new regions were also having other issues we've been trying to work around first (dns configuration strangeness, not enough cpu:ram ratio, high error rate on nova boot calls) | 14:50 |
| fungi | and rax also just started offering glance as well, though you have to do "special things" apparently to be able to make it work there at the moment | 14:51 |
| *** basha has quit IRC | 14:51 | |
| Kiall | Yea that - dns config one annoys me too, but it's a neutron issue :'( | 14:51 |
| fungi | so we're getting closer to being able to start looking at switching how nodepool does image management for the bette | 14:51 |
| fungi | r | 14:51 |
| *** gokrokve has quit IRC | 14:51 | |
| *** gokrokve has joined #openstack-infra | 14:53 | |
| BobBall | I've got a strange issue in RAX with the introduction of trove... http://paste.openstack.org/show/74753/ - I thought that localhost is being resolved to the v6 address yet everything is configured to use v4 - but preferring v4 didn't solve it. | 14:56 |
| BobBall | Actual error is when accessing localhost: "Caused by <class 'socket.gaierror'>: [Errno -9] Address family for hostname not supported" | 14:56 |
| *** thedodd has joined #openstack-infra | 14:56 | |
| *** thuc has joined #openstack-infra | 15:00 | |
| fifieldt | lol@ freenode+ | 15:00 |
| *** thuc has quit IRC | 15:01 | |
| ttx | fifieldt: that would make the DefCore meetings much more integrated. | 15:01 |
| fifieldt | haha | 15:01 |
| fifieldt | touche! | 15:01 |
| *** thuc has joined #openstack-infra | 15:02 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: normalize the services for a post grizzly world https://review.openstack.org/84287 | 15:05 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: turn back on log enforcement https://review.openstack.org/81216 | 15:05 |
| sdague | BobBall: how about that https://review.openstack.org/81216 for 3rd party? | 15:05 |
| *** zehicle_at_dell has joined #openstack-infra | 15:06 | |
| BobBall | Perfect sdague - thanks. | 15:08 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: turn off clean log enforcement in stress tests https://review.openstack.org/84469 | 15:08 |
| jeblair | fungi: good morning | 15:09 |
| fungi | jeblair: good morning to you | 15:09 |
| fungi | jeblair: not sure if you caught the scrollback, but while i was at dinner last night clarkb ended up restarting nodepool and discovered that the subnode changes were failing to write into the nonexistent /etc/nodepool dir on new nodes | 15:10 |
| sdague | jeblair / fungi: because this will change how a few things run in d-g jobs, https://review.openstack.org/#/c/84146/ would be good to make sure infra folks are on board with | 15:11 |
| sdague | dhellmann: actually your thoughts on https://review.openstack.org/#/c/84146/ would be welcome as well, I think it gets us out of the olso vs. pip issue once and for all | 15:11 |
| jeblair | fungi: ah yeah, that's created by the image build step; so i think we need to kick off image builds when we start with that version | 15:11 |
| fungi | he proposed a change to solve that, but in the interim rolled back the local clone on nodepool.o.o to the point where labels were introduced and then temporarily stopped puppet so we could consult with you | 15:11 |
| mordred | sdague: it looks good to me - do you forsee any particular effects in the gate? | 15:12 |
| *** chandankumar_ has joined #openstack-infra | 15:12 | |
| mordred | sdague: (other than less breakage) | 15:12 |
| mordred | sdague: it LOOKS like it will be a no-op for us | 15:12 |
| fungi | okay, so we roll those back forward, restart nodepoold and start new image builds immediately? | 15:13 |
| sdague | mordred: I don't | 15:13 |
| jeblair | fungi: yeah, that change won't work -- the user doesn't (necessarily) have permission to create that directory (it gets created world-writable) | 15:13 |
| sdague | just the old instincts on the breaks will change | 15:13 |
| sdague | because installing things from /opt into the system path will be a different behaviro | 15:14 |
| fungi | jeblair: how would you want to handle, e.g., being unable to get new images reliably in hpcloud-az2 for a while when we do that? | 15:14 |
| jeblair | fungi: so yes, that's probably the safest thing to do, though it'll slow new node creation for a bit. we could merge a change that avoids writing to that directory and make this a two-parter | 15:14 |
| sdague | I'm hoping it just makes all things better | 15:14 |
| *** erw has joined #openstack-infra | 15:14 | |
| *** dkliban is now known as dkliban_afk | 15:15 | |
| fungi | jeblair: oh, good point, if we can get in the image build change first and then the subnode change which uses it, we have the luxury of getting image updates in some of those problem providers on longer time scales | 15:15 |
| BobBall | Why in stack.sh is RABBIT_HOST defaulted to localhost rather than SERVICE_HOST? http://git.openstack.org/cgit/openstack-dev/devstack/tree/stack.sh#n428 | 15:18 |
| *** jcoufal has quit IRC | 15:18 | |
| sdague | BobBall: I expect to get around some failure we found previously | 15:19 |
| *** hashar_ has joined #openstack-infra | 15:19 | |
| BobBall | hmmmm... the use of "localhost" is the eventual cause of my trove ipv4/ipv6 issues... of course I can just specify RABBIT_HOST in my localrc but if there is a more general fix I wanted to go that route | 15:20 |
| sdague | actually this seems to go away back | 15:22 |
| *** hashar has quit IRC | 15:22 | |
| *** ildikov_ has quit IRC | 15:22 | |
| *** CaptTofu has joined #openstack-infra | 15:22 | |
| *** zns has quit IRC | 15:22 | |
| *** alexpilotti has quit IRC | 15:22 | |
| sdague | BobBall: if you want to propose a change, it's probably fair to | 15:23 |
| *** alff_ has quit IRC | 15:23 | |
| *** alff has quit IRC | 15:23 | |
| fungi | jeblair: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/trigger-job.py#n46 is no longer really a sane url... any ideas on how we should be identifying which merger to use when triggering? | 15:25 |
| *** markwash has quit IRC | 15:25 | |
| *** markwash has joined #openstack-infra | 15:26 | |
| *** hashar_ has quit IRC | 15:26 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a multinode label to nodepool https://review.openstack.org/84473 | 15:27 |
| jeblair | fungi: if retriggering, then use the value in ZUUL_URL that was previously passed | 15:28 |
| fungi | jeblair: ahh, right-o | 15:28 |
| jeblair | fungi: it should show up as a param | 15:28 |
| fungi | it does... how log will it take those to get gc'd? (or will they?) | 15:28 |
| fungi | s/log/long/ | 15:28 |
| *** zehicle_at_dell has quit IRC | 15:30 | |
| *** dstanek has quit IRC | 15:30 | |
| *** zehicle_at_dell has joined #openstack-infra | 15:31 | |
| BobBall | sdague: https://review.openstack.org/#/c/84475/ FYI | 15:31 |
| *** hashar has joined #openstack-infra | 15:33 | |
| jeblair | fungi: they should be there until we rm-rf the underlying git repos | 15:34 |
| fungi | jeblair: perfect | 15:34 |
| mordred | zaro: so, if you're going to start hacking on gerrit ... | 15:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a multinode label to nodepool https://review.openstack.org/84473 | 15:35 |
| mordred | zaro: I'd love it if I could request that gerrit attach patches or inline diffs into emails it sends me :) | 15:35 |
| *** mrodden has quit IRC | 15:35 | |
| jeblair | mordred: i think the new change screen is terrible from a usability and design pov. i don't think we can in good concience upgrade to a version that requires it. should addressing that be a priority? | 15:37 |
| mordred | jeblair: yah. prolly so | 15:38 |
| sdague | jeblair: ++ | 15:38 |
| mordred | zaro: as usual, NEVER listen to me when I drop suggestions for things to work on into irc | 15:38 |
| sdague | heh | 15:38 |
| mordred | jeblair: do we know a timeline for 2.9 yet? | 15:39 |
| jeblair | mordred: i don't know | 15:39 |
| sdague | mordred: honestly, I was sort of thinking once we have a version of gerrit that has a REST api, to do the notification piece completely out of band | 15:39 |
| mordred | jeblair: and/or is making a plan for addressing the new screen a thing we dan do at the summit? | 15:39 |
| sdague | because the filters available for notification are kind of limitted anyway | 15:40 |
| mordred | sdague: I think there are _many_ opportunities that open up once we have a REST api | 15:40 |
| sdague | agreed | 15:40 |
| mordred | sdague: but for notifications, if we wanted non-internal notifies... | 15:40 |
| *** yassine has quit IRC | 15:40 | |
| *** mgagne has joined #openstack-infra | 15:40 | |
| *** dizquierdo has quit IRC | 15:40 | |
| mordred | I think the ssh event-stream is probably a better place to get info to do that | 15:40 |
| sdague | possibly | 15:40 |
| mordred | like, we could do that now - the biggest problem would be how users change their notification settings | 15:41 |
| mordred | because I don't think code-review-per-dev-per-setting is a good idea :) | 15:41 |
| *** mfer has joined #openstack-infra | 15:41 | |
| jeblair | mordred: i'm shocked to hear you say that! | 15:41 |
| mordred | jeblair: I'm a shocking guy | 15:41 |
| jeblair | sdague: can you give me a really short summary of why non-editable makes a difference for the oslo libs? | 15:42 |
| sdague | because if you pip install -e /opt/stack/oslo.config | 15:42 |
| mordred | jeblair: aparently python namespace packages and pip and editable break because $reasons | 15:42 |
| sdague | then pip install olso.vmware | 15:42 |
| sdague | oslo.config is now gone | 15:42 |
| mordred | sdague: oh - you know what - that's the most helpful description I've heard of the problem | 15:43 |
| mordred | that actually makes sense to me why it breaks | 15:43 |
| fungi | jeblair: mordred: zaro: are we also stuck not being able to use the wip workflow feature effectively without using the new change screen, or was i misunderstanding that discussion about the pop-up pane for workflow only working in the new change screen? | 15:43 |
| fungi | is that a 2.9-only concern for now? | 15:43 |
| sdague | mordred: well, we only hit that issue yesterday :) | 15:43 |
| sdague | and I agree, it clarifies | 15:43 |
| mordred | sdague: but in general - develop/pip -e works by making symlinks to places /python .pth files - namespace pacakges use similar magic hacks | 15:44 |
| mordred | the two together ... oy | 15:44 |
| sdague | mordred: yep | 15:44 |
| *** zns has joined #openstack-infra | 15:45 | |
| mordred | sdague: have we considered just finishing the job and getting rid of our use of namespace packages? I'm not sure they REALLY actually get us anything... | 15:45 |
| sdague | mordred: I keep asking that one :) | 15:45 |
| mordred | I mean, 'it looks nicer' is all well and good - but there isn't an actual technical benefit | 15:46 |
| sdague | however, I think the use case for pip install -e for libraries in devstack isn't sound anyway | 15:46 |
| sdague | pip install -e /opt/stack/nova makes a ton of sense | 15:46 |
| mordred | sdague: sure | 15:46 |
| jeblair | sdague, mordred: thank you both very much for that. a lot of things make sense now (at least, i have a working knowledge) | 15:46 |
| sdague | jeblair: no problem | 15:46 |
| *** dprince has quit IRC | 15:46 | |
| mordred | jeblair: it was helpful for me too | 15:46 |
| *** thuc_ has joined #openstack-infra | 15:47 | |
| fungi | i also found it elucidating | 15:48 |
| sdague | jeblair: want to +A it - https://review.openstack.org/#/c/84146/ | 15:48 |
| fungi | i had never really thought much about how setup.py develop worked before | 15:48 |
| *** jcoufal has joined #openstack-infra | 15:48 | |
| *** UtahDave has joined #openstack-infra | 15:48 | |
| sdague | I think all parties have weighed in now on it | 15:48 |
| jeblair | sdague: will do | 15:49 |
| sdague | thanks | 15:49 |
| *** gyee has joined #openstack-infra | 15:49 | |
| pleia2 | good morning | 15:49 |
| sdague | completely unrelated, is infra using tagmail anywhere? | 15:49 |
| sdague | for puppet things | 15:49 |
| *** thuc has quit IRC | 15:50 | |
| anteaya | morning pleia2 | 15:50 |
| jeblair | sdague: does that cells-full failure look okay? | 15:51 |
| *** thuc_ has quit IRC | 15:51 | |
| sdague | the non voting one? | 15:51 |
| *** mrodden has joined #openstack-infra | 15:51 | |
| jeblair | sdague: yep | 15:51 |
| sdague | yeh, that's not supposed to pass yet | 15:51 |
| jeblair | cool | 15:51 |
| dstufft | mordred: ~magic hacks~ | 15:51 |
| dstufft | mordred: FWIW +100 on killing namespace packages | 15:52 |
| dstufft | mordred: oslo_foo isn't inheriently worse than oslo.config, but it'll work more consistently :) | 15:52 |
| mordred | dstufft: ++ | 15:53 |
| mordred | sdague: we do not | 15:54 |
| mordred | sdague: we try to avoid email notification for things if we can :) | 15:54 |
| cody-somerville | If one changes the min-ram value in nodepool, do you have to do anything special to deploy that change? | 15:54 |
| sdague | mordred: no worries, just curious. I hacked together a gpg version of it this weekend for personal use, was trying to figure out if it was of interest | 15:55 |
| *** julienvey has quit IRC | 15:55 | |
| mordred | sdague: I think too many of our puppet configs are too chatty (stuff that erroneously thinks it's changed every time) for it to be of use | 15:56 |
| fungi | yeah, idempotent exec blocks are a problem there | 15:57 |
| fungi | they run on every update and that shows as a change | 15:57 |
| jeblair | cody-somerville: theoretically, no, but if there's a bug it may need to be restarted. also, depending on in what manner you changed it and the behavior of your cloud, launching new nodes from an image of a different size may or may not work. | 15:57 |
| sdague | fungi: sure, you definitely need to make the idempotent exec blocks have a creates or unless clause | 15:57 |
| *** jlibosva has quit IRC | 15:57 | |
| fungi | sdague: right, we should totally do that. we haven't (yet) | 15:58 |
| sdague | my config is only about 6 - 8 nodes | 15:58 |
| sdague | and pretty clean on runs, so it's handy to see when things update | 15:58 |
| cody-somerville | jeblair: Will it delete and recreate nodes that no longer meet min-ram requirement? | 15:58 |
| jeblair | cody-somerville: no | 15:59 |
| *** changbl has joined #openstack-infra | 15:59 | |
| *** jgallard has quit IRC | 16:00 | |
| *** hashar has quit IRC | 16:00 | |
| *** lcheng has joined #openstack-infra | 16:01 | |
| fungi | zaro: i was finally able to get my https://review.openstack.org/82583 fix for ggp onto enough nodepool images that retriggering each of the tag jobs for your presentation hit a node with the updated script and worked, so it's showing up in the index now | 16:01 |
| *** hashar has joined #openstack-infra | 16:01 | |
| *** moted has quit IRC | 16:01 | |
| *** hashar has quit IRC | 16:02 | |
| *** moted has joined #openstack-infra | 16:02 | |
| jeblair | nibalizer: regarding https://review.openstack.org/#/c/84356/1 may i suggest the openstack-dev/sandbox repo? :) | 16:03 |
| *** alff has joined #openstack-infra | 16:03 | |
| *** zehicle_at_dell has quit IRC | 16:03 | |
| *** BadCub01 has quit IRC | 16:05 | |
| *** jlibosva has joined #openstack-infra | 16:05 | |
| *** jgallard has joined #openstack-infra | 16:08 | |
| *** e0ne has quit IRC | 16:09 | |
| *** amotoki has joined #openstack-infra | 16:10 | |
| *** Sukhdev has joined #openstack-infra | 16:10 | |
| *** jepoy has quit IRC | 16:12 | |
| *** mwagner_zzz has joined #openstack-infra | 16:16 | |
| *** jgallard has joined #openstack-infra | 16:17 | |
| *** alff_ has joined #openstack-infra | 16:17 | |
| *** thuc has joined #openstack-infra | 16:18 | |
| *** thuc has quit IRC | 16:18 | |
| *** thuc has joined #openstack-infra | 16:19 | |
| zaro | mordred: | 16:19 |
| zaro | morning | 16:19 |
| zaro | fungi: thanks | 16:20 |
| *** mfer has quit IRC | 16:20 | |
| openstackgerrit | Eric Windisch proposed a change to openstack-infra/config: Skip requirements check for Docker https://review.openstack.org/84488 | 16:22 |
| jogo | clarkb: ahh that explains the spike on http://jogo.github.io/gate/ | 16:23 |
| *** jgallard has quit IRC | 16:24 | |
| sdague | jogo: what is the explanation? | 16:24 |
| *** CaptTofu has quit IRC | 16:24 | |
| jogo | sdague: https://review.openstack.org/#/c/84319/ | 16:25 |
| sdague | oh, interesting | 16:25 |
| jogo | graphite shows things are fixed again | 16:25 |
| sdague | ok, so that was the big spike | 16:26 |
| jogo | heh yeah, that would result in a 100% failure | 16:26 |
| jogo | the graph hides that in its moving averaging | 16:26 |
| sdague | so how come that triggered the network bugs? | 16:26 |
| sdague | http://status.openstack.org/elastic-recheck/ | 16:27 |
| jogo | sdague: well large-ops doesn't create VMs | 16:27 |
| sdague | I didn't think they ran real ssh | 16:27 |
| sdague | ok, so how are the ssh tests failing at all? | 16:27 |
| sdague | because they shouldn't be run, right? | 16:27 |
| fungi | i believe the problem is that they weren't supposed to, and that's where it was breaking | 16:27 |
| jogo | sdague: it ran ssh by accident or something | 16:27 |
| fungi | yeah, because that comparison was missed, tests tried to ssh to not-real virtual machines | 16:28 |
| sdague | ok | 16:28 |
| jogo | sdague: it hit the else statement | 16:28 |
| jogo | which triggered smoke tests which then failed | 16:28 |
| sdague | and d-g doesn't gate on large-ops jobs? | 16:28 |
| fungi | the end result, of course, looks a lot like the bug where the instance doesn't come up and the ssh test legitimately fails | 16:29 |
| sdague | yep | 16:29 |
| *** dtroyer has quit IRC | 16:29 | |
| sdague | just trying to sort out if this is a hole we need to fill | 16:29 |
| zaro | fungi, mordred : I believe WIP will work on old change screen. from my discussion with _david_ what doesn't work is that when you set/unset WIP the popup to provide a message does not work. | 16:29 |
| jogo | sdague: hmm good question | 16:29 |
| nibalizer | jeblair: okay, next time i'll use that one | 16:29 |
| fungi | zaro: ahh, okay. so the instructions which go along with the workflow label aren't displayed? | 16:30 |
| jogo | sdague: devstack gate does gate on large-ops | 16:30 |
| zaro | fungi, mordred : the change does get set into the correct state, but user just doen't get to enter a message for the state change. | 16:30 |
| *** dtroyer has joined #openstack-infra | 16:30 | |
| sdague | jogo: that seems like something we probably want to change | 16:31 |
| *** zns has quit IRC | 16:31 | |
| fungi | zaro: oh! the comment box. okay | 16:31 |
| zaro | fungi, mordred : yes, comment box. i assume if you wanted to add a comment you would have to do it in two steps. | 16:31 |
| jogo | sdague: ohh I think I see it | 16:32 |
| *** maru_afk is now known as marun | 16:32 | |
| jogo | sdague: when https://review.openstack.org/#/c/80643/ merged the large-ops value was 1 | 16:32 |
| fungi | zaro: that seems like something we could live with i suppose | 16:32 |
| *** UtahDave has quit IRC | 16:32 | |
| *** dteselkin_ has joined #openstack-infra | 16:32 | |
| zaro | jeblair: your thoughts? ^ | 16:33 |
| jogo | sdague: then it got switched to 50 | 16:33 |
| jogo | as we want to turn that from a bool (0,1) to the number of instances to boot | 16:33 |
| fungi | jogo: sdague: so the hole where we didn't gate on the actual bug trigger was the change to the jjb definition | 16:33 |
| *** Sukhdev has quit IRC | 16:33 | |
| sdague | ah, so it was layout change | 16:33 |
| sdague | which isn't currently gateable | 16:34 |
| jogo | sdague: correct I98baec6e0890f6b10afbd9858ded933f59db9725 | 16:34 |
| jogo | in config | 16:34 |
| sdague | you know, that would be a kind of awesome thing to do, actually run the results of layout job changes | 16:34 |
| fungi | that's something else which might be more solvable in a post-jenkins world... testing the implications of alterations to the definitions of existing jobs | 16:34 |
| sdague | but it would be complicated | 16:34 |
| *** zns has joined #openstack-infra | 16:34 | |
| sdague | fungi: yep | 16:35 |
| jogo | my limited understanding of this, is this would be a side effect of gozer. as they will need to produce code to automatically spin up an infra | 16:35 |
| dhellmann | sdague: we talked about that the other day -- I'm ok with it, and we'll see if it causes issues or confusions for devs working on those libs. | 16:35 |
| dhellmann | sdague: were you still seeing ns package issues? | 16:36 |
| zaro | mordred: i've been exposed! i'll keep your gerrit request on the peripheral. i think i saw an openstack-ci bug on that already. | 16:36 |
| zaro | mordred: been assigning gerrit bugs to myself, but really just wanted to star them but didn't see a way to do that in LP. | 16:37 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Protect against /etc/nodepool not existing https://review.openstack.org/84492 | 16:40 |
| jeblair | fungi: ^ (testing that took a while) | 16:41 |
| fungi | jeblair: oh! awesome | 16:41 |
| *** dprince has joined #openstack-infra | 16:41 | |
| jeblair | fungi: (i chose the provider that is slow to spin up nodes as opposed to the one that has trouble building images) | 16:41 |
| fungi | jeblair: yep, that looks like just the ticket | 16:41 |
| fungi | jeblair: hah | 16:42 |
| fungi | jeblair: it's awesome that we have such a choice | 16:42 |
| *** gothicmindfood has joined #openstack-infra | 16:43 | |
| dansmith | clarkb: doh, sorry about that largeops omission | 16:43 |
| *** derekh has quit IRC | 16:44 | |
| *** dstanek has joined #openstack-infra | 16:50 | |
| fungi | jeblair: are the 'No handlers could be found for logger "nodepool.NodePool"' errors during test_subnodes on your radar? | 16:53 |
| *** mihgen has quit IRC | 16:54 | |
| *** mriedem has quit IRC | 16:56 | |
| fungi | oh, actually that was on a python26 jobm but now i;m looking at a python27 job hitting something else... | 16:56 |
| *** mihgen has joined #openstack-infra | 16:57 | |
| *** mihgen has quit IRC | 16:57 | |
| *** BadCub01 has joined #openstack-infra | 16:57 | |
| *** khyati has joined #openstack-infra | 16:58 | |
| fungi | "AttributeError: 'NoneType' object has no attribute 'db'" (not sure whether that's a cause or an effect yet) | 16:58 |
| fungi | self.config is ending up as None | 16:58 |
| *** mgagne has quit IRC | 16:59 | |
| fungi | ahh, yep they're the same... just the python26 job ends up with the traceback in the subunit log rather than the console | 17:00 |
| *** zehicle_at_dell has joined #openstack-infra | 17:00 | |
| *** SumitNaiksatam has quit IRC | 17:00 | |
| *** arosen has quit IRC | 17:02 | |
| *** jcoufal has quit IRC | 17:04 | |
| *** imcsk8_lap has quit IRC | 17:04 | |
| *** imcsk8_lap has joined #openstack-infra | 17:04 | |
| *** afazekas is now known as afazekas_noteboo | 17:05 | |
| *** amotoki has quit IRC | 17:05 | |
| *** Sukhdev has joined #openstack-infra | 17:05 | |
| *** CaptTofu has joined #openstack-infra | 17:05 | |
| *** Ryan_Lane has joined #openstack-infra | 17:08 | |
| *** afazekas has joined #openstack-infra | 17:08 | |
| *** Sukhdev has quit IRC | 17:09 | |
| *** UtahDave has joined #openstack-infra | 17:10 | |
| clarkb | morning | 17:11 |
| *** sarob has joined #openstack-infra | 17:12 | |
| *** harlowja_away is now known as harlowja | 17:14 | |
| anteaya | morning clarkb | 17:14 |
| fungi | clarkb: see the immediate above, i think we may have an issue with the unit tests for nodepool (in addition to the tox/sphinx issue you spotted last night). also getting 84492 in so we could restart puppet on nodepool.o.o (i can arrange a nodepool restart later in the evening so we're sure it's sane) would be stellar | 17:14 |
| *** mwagner_zzz is now known as mwagner_ | 17:14 | |
| fungi | i'm poking at the nodepool unit test failures now | 17:14 |
| clarkb | fungi: yup, I didn't have time to look into the unit test failures last night | 17:14 |
| clarkb | fungi: I am guessing some small minor detail issue difference between jeblair's local test setup and the slaves | 17:15 |
| clarkb | fungi: it looked like jobs were timing out at the partial subunit log was being captured though | 17:15 |
| fungi | clarkb: yeah, i think it's not getting a config object | 17:15 |
| *** SumitNaiksatam has joined #openstack-infra | 17:15 | |
| fungi | self.config ends up being None for the pool | 17:16 |
| *** mgagne has joined #openstack-infra | 17:17 | |
| *** david-lyle is now known as david-lyle_afk | 17:17 | |
| *** jpich has quit IRC | 17:18 | |
| clarkb | interesting | 17:19 |
| *** mriedem has joined #openstack-infra | 17:19 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: ngDelete directive https://review.openstack.org/84504 | 17:20 |
| *** jlibosva has quit IRC | 17:20 | |
| fungi | i wonder if we need to call pool.loadConfig() before pool.getDB | 17:21 |
| fungi | or maybe self.updateConfig() | 17:23 |
| *** alexpilotti has joined #openstack-infra | 17:25 | |
| *** afazekas_noteboo is now known as _afazekas | 17:25 | |
| jeblair | fungi, clarkb: oh, do you have a link to a completed test run? | 17:25 |
| fungi | jeblair: http://logs.openstack.org/27/84327/1/check/gate-nodepool-python27/41c96ba/console.html | 17:26 |
| fungi | i've got a change on my way to test the theory | 17:26 |
| fungi | but if you know off the top of your head what's missing, all's the better | 17:26 |
| jeblair | fungi: not off the top | 17:26 |
| *** zhiyan is now known as zhiyan_ | 17:27 | |
| jeblair | fungi: maybe a race | 17:27 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Update config before accessing DB in tests https://review.openstack.org/84505 | 17:27 |
| fungi | may not have the method order correct on that either | 17:27 |
| jeblair | fungi: wow, you'd think a 2 second sleep would be enough. yay python thread scheduling? | 17:28 |
| jeblair | fungi: tripleo looks :( | 17:29 |
| *** thedodd has quit IRC | 17:29 | |
| jeblair | is there a known issue or should i investigate? | 17:29 |
| fungi | jeblair: issue is not known to me, if it's known | 17:30 |
| jeblair | fungi, clarkb: so the nodepool docs error looks like a sphinx version issue we've seen before: https://jenkins03.openstack.org/job/gate-nodepool-docs/2/console | 17:30 |
| *** morganfainberg_Z is now known as morganfainberg | 17:31 | |
| jeblair | i don't understand why we would hit it though, i think we're using the same version as the other infra tools | 17:31 |
| fungi | jeblair: looking at test_node and test_subnodes, should we be calling pool.startup() instead of pool.start() maybe? | 17:31 |
| *** CaptTofu has quit IRC | 17:32 | |
| fungi | jeblair: the docs error clarkb has a change to the tox.ini which fixes it | 17:32 |
| *** CaptTofu has joined #openstack-infra | 17:32 | |
| jeblair | fungi: no -- .start() starts the thread (which calls the run method which calls startup) | 17:32 |
| fungi | jeblair: https://review.openstack.org/84327 | 17:33 |
| jeblair | fungi, clarkb: oooooh. thanks. :) | 17:33 |
| fungi | jeblair: oh, okay. i was having trouble tracking down where start() was defined | 17:33 |
| jeblair | fungi: threading.Thread.start() | 17:33 |
| fungi | yep. yay inheritance | 17:34 |
| jogo | what is the status of python-openstackclient? | 17:34 |
| jogo | its not part of a program | 17:34 |
| mordred | jogo: that is its status | 17:34 |
| mordred | jogo: dtroyer may apply for a program for it | 17:34 |
| jogo | mordred: then it belongs in stackforge | 17:34 |
| jogo | ? | 17:34 |
| mordred | jogo: we had that discussion, which led to the idea that dtroyer needed to apply for a program potentially | 17:35 |
| jogo | and we shouldn't be deprecating other CLIs and say use openstackcl;ient | 17:35 |
| fungi | jogo: except we'd rather not go to the trouble of moving it twice in a short period of time | 17:35 |
| jogo | fungi: ahh | 17:35 |
| jogo | FWIW I am all for it | 17:35 |
| *** sarob has quit IRC | 17:35 | |
| mordred | jogo: I think the idae is that, as a project, we want this openstackclient thing - but we haven't gotten our ducks in a row about it yet | 17:35 |
| *** ildikov_ has joined #openstack-infra | 17:35 | |
| mordred | so I think we're trying to figure out how to properly indicate that everytone actually does want the thing :) | 17:35 |
| *** sarob has joined #openstack-infra | 17:36 | |
| jogo | mordred: right, so the question is: keystoneclient doesn't have a v3 CLI instead they say use openstackclient | 17:36 |
| jogo | mordred: which is really awesome, but IMHO premature | 17:36 |
| jeblair | fungi: topic in tripleo says 'firedrill ci overcloud down' so i'm assuming it's known | 17:36 |
| *** mfer has joined #openstack-infra | 17:36 | |
| jeblair | fungi: we're getting ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-b74e7625-1bb2-43e1-a456-6159760a2c52) | 17:36 |
| fungi | jeblair: okay, thanks! | 17:36 |
| dtroyer | fwiw, I should have the program application shortly…still working out a few details WRT other projects that may be a part of it | 17:36 |
| *** che-arne has quit IRC | 17:36 | |
| *** dkliban_afk is now known as dkliban | 17:36 | |
| *** CaptTofu has quit IRC | 17:37 | |
| jogo | dtroyer: awesome, so I have a hunch that with the other CLI related efforts that program application will have to cover a few stackforge repos as welll | 17:37 |
| jogo | dtroyer: thoughts on if it makes sense for 'If you mean the keystone CLI that is currently bundled as part of the | 17:37 |
| *** mfer has quit IRC | 17:37 | |
| jogo | python-keystoneclient then yes that is deprecated in favour of | 17:37 |
| jogo | python-openstackclient. | 17:37 |
| jogo | ' | 17:37 |
| mordred | jogo: which stackforge projects? | 17:38 |
| * jogo checks | 17:38 | |
| dtroyer | jogo: I've been supportive of that since dolphm started saying it a while back…the Identity stuff is the most completely implemented set of commands in OSC of any of the APIs | 17:39 |
| jogo | stackforge/python-openstacksdk | 17:39 |
| * mordred is in favor of our new dtroyer overlord | 17:39 | |
| mordred | jogo: I think that may or may not be a related thing, depending on what dtroyer pitches the scope of openstackclient to be | 17:39 |
| jogo | mordred: perhaps | 17:40 |
| dtroyer | The SDK projects are likely not going to be part of the initial program application based on what I know ATM, they will follow when they are ready... | 17:40 |
| mordred | jogo: the sdk is intended to be like openstackclient - except to be a single lib for devs to use | 17:40 |
| jeblair | yaaaaaaaaaaaaaaaaaaaaaaay | 17:40 |
| mordred | yah. I think it would be great to have both things - but I think the sdk is a bit younger | 17:40 |
| dtroyer | the python-openstackclinet will eventually be the bottom half of OSC... | 17:40 |
| jogo | dtroyer: I guess you can always ammend the charter for the program | 17:40 |
| dtroyer | rats….the python-openstacksdk will ... | 17:40 |
| mordred | dtroyer: ++ | 17:40 |
| *** e0ne has joined #openstack-infra | 17:40 | |
| jogo | I think a client focused prgram would be neat | 17:40 |
| *** sarob has quit IRC | 17:40 | |
| notmyname | what version of pep8 are we using now? | 17:40 |
| mordred | notmyname: 1.4.5 | 17:41 |
| jogo | notmyname: http://git.openstack.org/cgit/openstack-dev/hacking/tree/requirements.txt?h=0.8.x | 17:41 |
| mordred | jogo's answer is better | 17:41 |
| jogo | dtroyer: so I think we have a chicken and egg issue with you answer | 17:41 |
| jogo | re: use OSC | 17:41 |
| dtroyer | no, today OSC uses the existing client libs…when its ready, it'll be using the SDK | 17:42 |
| jogo | dtroyer: well from a governance point of view I mean. | 17:42 |
| anteaya | dtroyer: while we are talking programs and such, do you think devstack can rustle up a ptl candidate? | 17:42 |
| dtroyer | I expect that'll be some time based on how fast things seem to be going in this area | 17:42 |
| dtroyer | ah, well, we don't use them today | 17:43 |
| jogo | dtroyer: a program shouldn't be able to depreacte something in favor of something that *should* be in stackforge | 17:43 |
| dtroyer | and I expect that python-openstacksdk will be part of the program before we start shipping an OSc that uses it | 17:43 |
| * anteaya notes that 8 programs still need at least one ptl candidate | 17:43 | |
| dtroyer | anteaya: noted…*its in the mail* | 17:43 |
| anteaya | dtroyer: thank you | 17:44 |
| jogo | anteaya: whoa what programs? | 17:44 |
| anteaya | https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 | 17:44 |
| mtreinish | jogo: you going to step up? | 17:44 |
| jogo | dtroyer: one solution to this is to sort out the status of OSC soon | 17:44 |
| anteaya | the ones with no candidates | 17:44 |
| dtroyer | that lazy devstack ptl for one hasn't sent anything yet | 17:44 |
| dtroyer | jogo: right…hence the pending program app for it | 17:44 |
| jogo | mtreinish: oh I should do keystone ;) | 17:44 |
| jogo | dtroyer: oh its pending? | 17:45 |
| anteaya | at least one commit in the last two releases gets your hat in the ring | 17:45 |
| *** erw has quit IRC | 17:45 | |
| dtroyer | well, pending in the sense that I'm going to send it in a day or three…I'm still getting everything put together | 17:45 |
| anteaya | fair enough | 17:45 |
| jogo | dtroyer: ahh cool | 17:46 |
| jogo | anteaya: I do have a commit in keystone heh | 17:46 |
| anteaya | I'm going to send out a nagging email tonight | 17:46 |
| sdague | dtroyer: have you given devstack a run on 13.10 recently? | 17:46 |
| anteaya | jogo: send an email | 17:46 |
| jogo | haha | 17:46 |
| *** chandankumar_ has quit IRC | 17:46 | |
| dtroyer | sdague: no, I'm stuck in precise-land for now | 17:46 |
| mtreinish | jogo: you've got my vote (except I don't think I have any keystone commits :) ) | 17:46 |
| anteaya | mtreinish: there's still time | 17:46 |
| sdague | so, it definitely doesn't work from my recent experiences | 17:47 |
| mtreinish | anteaya: that was my intent with: https://review.openstack.org/#/c/84258/ :) | 17:47 |
| sdague | which concerns me about 14.04 | 17:47 |
| anteaya | mtreinish: why not, pressure some cores, get yourself a vote | 17:48 |
| mtreinish | anteaya: actually would python-keystoneclient count so I could vote for jogo? | 17:49 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Tag Input Control https://review.openstack.org/83945 | 17:50 |
| jogo | mtreinish: woot! | 17:50 |
| *** amcrn has joined #openstack-infra | 17:51 | |
| *** jooools has quit IRC | 17:51 | |
| sdague | anteaya: is voting based on contributions in icehouse? or i & h? | 17:53 |
| anteaya | mtreinish: sure | 17:53 |
| anteaya | i & h | 17:53 |
| anteaya | and eligibiligy is based on time | 17:53 |
| anteaya | committers for one of the program projects over the Havana-Icehouse timeframe (April 4, 2013 06:00 UTC to April 4, 2014 05:59 UTC) | 17:54 |
| anteaya | so merges prior to April 4 05:50 utc count | 17:54 |
| anteaya | fire up those doc string patches | 17:54 |
| clarkb | sorry actually at computer now | 17:55 |
| clarkb | tripleo is sad because my by hand fix to the nodepool.yaml didn't bump the max servers. Looking at that now | 17:56 |
| clarkb | fixed, max is now 55 | 17:57 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update jenkins build-timeout plugin https://review.openstack.org/84512 | 18:00 |
| *** mtreinish has quit IRC | 18:00 | |
| fungi | clarkb: that doesn't seem like it would cause "ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)" | 18:00 |
| *** mwagner_ is now known as mwagner_afk | 18:00 | |
| clarkb | oh, is that the actual issue? | 18:00 |
| clarkb | I guess thats something we punt to #tripleo? | 18:00 |
| fungi | looks that way, and they have a firedrill topic up in their channel so we assume they're working to solve it | 18:01 |
| clarkb | ok | 18:01 |
| *** zns has quit IRC | 18:01 | |
| fungi | but can't hurt to check with them if you want to | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable sudo for rally-install job https://review.openstack.org/84394 | 18:01 |
| *** mtreinish has joined #openstack-infra | 18:01 | |
| *** jamielennox|away is now known as jamielennox | 18:02 | |
| clarkb | fungi: jeblair: anything else I can do to help debug the nodepool unit test trouble? | 18:03 |
| jeblair | hrm, i ran fungi's test locally and it works but it looks like the tests are still heading for a timeout | 18:04 |
| jeblair | i'll hold a node | 18:04 |
| fungi | yay, my loop of image build attempts for hpcloud-az2 bare-precise finally paid off! | 18:04 |
| *** julim has quit IRC | 18:04 | |
| fungi | yeah, i think that didn't seem to help matters | 18:04 |
| clarkb | my change will need to be squashed into whatever the unittest fix is, feel free to do that and abandon my change when we get there | 18:05 |
| fungi | i'm going to try shuffling the updateConfig() call to come after the start() call | 18:05 |
| sdague | https://review.openstack.org/#/c/81216/5 could use a +A before another service manages to break itself :) | 18:05 |
| fungi | sdague: that's safe now? | 18:05 |
| sdague | I believe so | 18:05 |
| jeblair | fungi: not sure i like that; could end up with concurrent modification in multiple threads | 18:05 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 18:05 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update jenkins-dev.o.o with same plugins as jenkins.o.o https://review.openstack.org/77040 | 18:05 |
| clarkb | sdague: looking | 18:06 |
| sdague | fungi: it's green across the board (nv included) | 18:06 |
| fungi | jeblair: you're probably right. i'm short-sighted when it comes to concurrency stuff | 18:06 |
| *** julim has joined #openstack-infra | 18:06 | |
| *** derekh has joined #openstack-infra | 18:07 | |
| fungi | sdague: last time we discovered that there were some errors which were existing already and cropped up infrequently but frequently enough to introduce some heavy gate reset frequency. should we recheck this a few times to be sure? | 18:07 |
| sdague | fungi: realistically, we've massively decreased the logs here | 18:07 |
| fungi | sdague: okay, sounds pretty safe in that case | 18:08 |
| sdague | so I'm feeling pretty good | 18:08 |
| sdague | we also were previously whitelisting lines in a log | 18:08 |
| *** LinuxJedi has joined #openstack-infra | 18:08 | |
| fungi | i'm +2 on it but i'll let clarkb approve | 18:08 |
| sdague | now we are whitelisting whole log files | 18:08 |
| fungi | that gives you a bigger hammer, agreed | 18:08 |
| clarkb | approved | 18:09 |
| sdague | woot | 18:09 |
| sdague | ok, now time for a bike ride | 18:09 |
| fungi | get a log file clean and then remove it from the whitelist, lather, rinse, repeat | 18:09 |
| sdague | as we've finally got our first sunny 60 degree day here | 18:09 |
| LinuxJedi | hey guys. So I'm trying to do "git log master..branch-X" on libra but it is showing stuff that has been cherry-picked from one to the other. Is there some magic to this I'm missing? | 18:09 |
| *** zns has joined #openstack-infra | 18:09 | |
| sdague | fungi: exactly | 18:09 |
| anteaya | sdague: woot enjoy your bike ride | 18:09 |
| jeblair | http://closedstack.org/ | 18:10 |
| fungi | LinuxJedi: i don't think there's an easy way to filter out cherry-picks other than maybe commit message matching... a c-p is just another commit from a git standpoint | 18:10 |
| LinuxJedi | shit... thanks anyway | 18:10 |
| *** Ajaeger has joined #openstack-infra | 18:10 | |
| fungi | it doesn't share the original's commit sha | 18:10 |
| clarkb | jeblair: nice | 18:11 |
| fungi | gets a new one of its own | 18:11 |
| fungi | teehee | 18:11 |
| fungi | "9:00 Doors Locked - Nobody may leave from this point forward." | 18:11 |
| *** jerryz has joined #openstack-infra | 18:11 | |
| *** sarob has joined #openstack-infra | 18:12 | |
| anteaya | nice logo too | 18:12 |
| *** dkranz has quit IRC | 18:12 | |
| *** sarob has quit IRC | 18:12 | |
| LinuxJedi | I'm not sure whether it will be faster to manually go through these 126 commits or script it.... Actually I don't work for cloud any more, I'll get the team that run it now to do it... | 18:12 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 18:12 |
| clarkb | LinuxJedi: protip, use the merge | 18:12 |
| fungi | LinuxJedi: :( we miss you! | 18:12 |
| *** dstufft has quit IRC | 18:13 | |
| fungi | i also very much like "address the needs of antsy, threatened IT departments worldwide" | 18:13 |
| *** ominakov has quit IRC | 18:13 | |
| *** _nadya_ has joined #openstack-infra | 18:13 | |
| LinuxJedi | clarkb: I'll take that in advisement if I ever run a project on Gerrit again :) | 18:13 |
| jogo | anteaya: ping | 18:13 |
| LinuxJedi | fungi: lol! You guys are doing fine without me :) | 18:14 |
| anteaya | here I am | 18:14 |
| *** dstufft has joined #openstack-infra | 18:14 | |
| *** Sukhdev has joined #openstack-infra | 18:14 | |
| jogo | how do I find the contact info for hpyer-v CI? | 18:14 |
| LinuxJedi | and I'll be at the Openstack Summit this year | 18:14 |
| fungi | LinuxJedi: find us. there will be beer | 18:14 |
| mordred | jogo: primeministerp should know | 18:14 |
| anteaya | jogo: https://review.openstack.org/#/admin/groups/91,members | 18:14 |
| LinuxJedi | I shall, hopefully I'll be able to talk about the things I'm working on at that point | 18:14 |
| jogo | primeministerp: its broken | 18:14 |
| anteaya | the email is listed | 18:14 |
| anteaya | jogo: do you have a link to the brokeness? | 18:15 |
| jogo | anteaya: perfect | 18:15 |
| jogo | anteaya: https://review.openstack.org/#/c/83752/ | 18:15 |
| jogo | http://64.119.130.115/83752/2/ | 18:15 |
| *** talluri has joined #openstack-infra | 18:15 | |
| anteaya | jogo: help me see the brokeness, so far I am not spotting it | 18:17 |
| clarkb | jeblair: for https://review.openstack.org/#/c/76057/4 I had a brilliant idea this morning. Would it be ok if I make TestScheduler's setUp() more modular, then introduce a second class that inherits from TestScheduler and sets gerrit.fetch_url and gerrit.poll? Then we would continue to test existing behavior and make sure that all of those tests work with the new behavior too. It will double the test | 18:18 |
| clarkb | count though | 18:18 |
| jogo | anteaya: that is the review to open up juno | 18:18 |
| clarkb | jeblair: its a bit of a hack to make it work, but is straightforward | 18:18 |
| jogo | and I went on the comments the systems added saying passed or failed | 18:18 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Remove py3 gate job from kite https://review.openstack.org/84372 | 18:18 |
| *** Ajaeger has quit IRC | 18:19 | |
| anteaya | hmmm, not sure what behaviour a ci system should have in that situation | 18:19 |
| anteaya | but build failed probably isn't it | 18:20 |
| jogo | anteaya: this is more useful http://www.rcbops.com/gerrit/reports/nova-cireport.html | 18:20 |
| fungi | jogo: its tempest log shows some failing tests, for sure | 18:20 |
| fungi | jogo: are you saying hyperv should work, or that it shouldn't report results which differ from upstream ci, or what? | 18:21 |
| *** thuc has quit IRC | 18:22 | |
| anteaya | yeah 100% failing probably needs some attention | 18:22 |
| jogo | fungi: that it should be passing | 18:22 |
| jogo | anteaya: exactyl | 18:22 |
| jogo | exactly | 18:22 |
| *** CaptTofu has joined #openstack-infra | 18:23 | |
| fungi | jogo: got it. is there a chance someone approved a change not long ago which breaks the hyperv backend and ignored a report from it? | 18:24 |
| *** thuc has joined #openstack-infra | 18:24 | |
| *** thuc has quit IRC | 18:24 | |
| fungi | at some point more than a week ago | 18:24 |
| *** thuc has joined #openstack-infra | 18:24 | |
| jogo | perhaps | 18:24 |
| *** vhoward- has left #openstack-infra | 18:25 | |
| jogo | asking the maintainers | 18:25 |
| *** dkranz has joined #openstack-infra | 18:26 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix tox's insane pip install command. https://review.openstack.org/84327 | 18:27 |
| jeblair | clarkb, fungi: ^ that's an attempt to fix all current nodepool test problems | 18:27 |
| fungi | aha! | 18:27 |
| jeblair | (i had a gearman server running locally) | 18:27 |
| fungi | that explains the connection failures on the gearman socket in the logs. i glossed over them because i figured they were a symptom | 18:28 |
| fungi | and my eyes jumped to the python traceback about the db connection | 18:28 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 18:32 |
| jeblair | clarkb, fungi: https://review.openstack.org/#/c/84327/ is ready | 18:32 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Protect against /etc/nodepool not existing https://review.openstack.org/84492 | 18:33 |
| jeblair | and ^ is rebased | 18:33 |
| *** erw has joined #openstack-infra | 18:34 | |
| clarkb | jeblair: is making the dir not the right solution? | 18:36 |
| clarkb | also I think it throws an IOError excepton if you want ot be more specific in the ignored error | 18:36 |
| jeblair | clarkb: the directory is created in the image build step; it didn't work yesterday because it was using old images | 18:37 |
| jeblair | clarkb: also, it raises "Exception" because it's the ssh client that does that (it raises Exception on any non-zero exit code of anything it runs) | 18:38 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update jenkins-dev.o.o with same plugins as jenkins.o.o https://review.openstack.org/77040 | 18:39 |
| clarkb | jeblair: right, but nodepool can create the dir | 18:39 |
| *** e0ne has quit IRC | 18:39 | |
| clarkb | jeblair: seems like nodepool should be responsible for that if it needs to write to that directory | 18:39 |
| jeblair | clarkb: nodepool _does_ create the dir -- in the image build | 18:39 |
| *** e0ne has joined #openstack-infra | 18:40 | |
| *** e0ne has quit IRC | 18:40 | |
| jeblair | clarkb: that's the only time it can create it because it might lose sudo after running the setup script | 18:40 |
| fungi | clarkb: this ties in with lifeless's comment yesterday about not assuming nodepool has root privs on the launched servers... it does on the template instance and sets /etc/nodepool 0x0777 at that point | 18:40 |
| *** e0ne has joined #openstack-infra | 18:40 | |
| jeblair | fungi: yup | 18:41 |
| clarkb | fungi: no lifeless' comment was about not assuming nodepool had root on the nodepool daemon server | 18:41 |
| fungi | oh, then i entirely misunderstood what he was talking about in that case | 18:41 |
| clarkb | fungi: he misunderstood what I was saying too :) | 18:42 |
| fungi | free misunderstandings for everyone! | 18:42 |
| jeblair | i was about to go read the logs, but it sounds like it might not be the best use of time. :) | 18:42 |
| clarkb | jeblair: is nodepool hardcoding the /etc/nodepool creation in code? or are users expected to know to create that dir? | 18:43 |
| clarkb | I think my biggest concern is whether or not users need to write special bash to make this work (and I don't remeber from reviews if nodepool does this regardless) | 18:43 |
| jeblair | clarkb: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n772 | 18:44 |
| clarkb | danke | 18:44 |
| jeblair | clarkb: it's documented too: http://docs-draft.openstack.org/27/84327/2/check/gate-nodepool-docs/e96a56e/doc/build/html/configuration.html#labels | 18:45 |
| clarkb | roger thanks | 18:46 |
| * clarkb reviews the change that deals with the exception | 18:46 | |
| jeblair | clarkb: and if you want to see the expected use: https://review.openstack.org/#/c/84473/ | 18:46 |
| clarkb | jeblair: comment on 84492 | 18:48 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix tox's insane pip install command. https://review.openstack.org/84327 | 18:48 |
| *** thedodd has joined #openstack-infra | 18:48 | |
| jeblair | clarkb: that change is intended to be temporary | 18:49 |
| jeblair | clarkb: i'll revert it in a day or so | 18:49 |
| jeblair | clarkb: so basically, i don't expect to actually care whether the code in that block runs during the time the try/except is there | 18:50 |
| openstackgerrit | Arnaud Legendre proposed a change to openstack/requirements: Update oslo.vmware to be >= 0.3 https://review.openstack.org/80120 | 18:54 |
| *** dstanek has quit IRC | 18:55 | |
| openstackgerrit | BK Box proposed a change to openstack-infra/config: Join gerritbot to the satori IRC channel https://review.openstack.org/79058 | 18:56 |
| *** jhesketh has joined #openstack-infra | 18:59 | |
| *** jcoufal has joined #openstack-infra | 18:59 | |
| *** jerryz has quit IRC | 18:59 | |
| jhesketh | Morning | 19:00 |
| SergeyLukjanov | meeting time> | 19:00 |
| anteaya | morning jhesketh | 19:01 |
| *** alff has quit IRC | 19:02 | |
| *** alff_ has quit IRC | 19:02 | |
| *** jcoufal has quit IRC | 19:03 | |
| *** jcoufal has joined #openstack-infra | 19:03 | |
| *** alff has joined #openstack-infra | 19:06 | |
| *** alff_ has joined #openstack-infra | 19:06 | |
| *** _nadya_ has quit IRC | 19:06 | |
| *** e0ne has quit IRC | 19:06 | |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** rlandy has quit IRC | 19:07 | |
| *** rlandy has joined #openstack-infra | 19:07 | |
| *** blamar has joined #openstack-infra | 19:08 | |
| *** markmcclain has quit IRC | 19:08 | |
| *** gokrokve_ has joined #openstack-infra | 19:10 | |
| *** dkranz has quit IRC | 19:11 | |
| *** thuc has quit IRC | 19:12 | |
| rcarrillocruz | hey guys, i need another +2 from a core reviewer for https://review.openstack.org/#/c/79205/ , could anyone please have a look? | 19:12 |
| *** thuc has joined #openstack-infra | 19:12 | |
| rcarrillocruz | clarkb: ^ , i put your suggestions, jeblair already gave the lgtm | 19:12 |
| *** ameade has quit IRC | 19:13 | |
| *** gokrokve has quit IRC | 19:14 | |
| *** gyee has quit IRC | 19:14 | |
| *** denis_makogon has joined #openstack-infra | 19:14 | |
| *** thuc has quit IRC | 19:17 | |
| *** talluri has quit IRC | 19:19 | |
| therve | fungi, If you can have a look at https://review.openstack.org/#/c/83761/ that'd be great | 19:19 |
| therve | jeblair, I think I handled your comments on that one too ^^ | 19:19 |
| kevinbenton | is there a known bug for this? | 19:20 |
| kevinbenton | http://logs.openstack.org/77/84377/2/gate/gate-neutron-python27/55e30bd/console.html | 19:20 |
| *** gokrokve_ has quit IRC | 19:20 | |
| kevinbenton | the parent patch has nothing to do with the components that are failing there | 19:21 |
| *** dkliban has quit IRC | 19:22 | |
| clarkb | kevinbenton: have you checked the rechecks page? http://status.openstack.org/rechecks/ | 19:22 |
| kevinbenton | clarkb: yeah, nothing obvious. that's why i came here | 19:22 |
| clarkb | kevinbenton: I would ask in the neutron channel | 19:23 |
| kevinbenton | clarkb: i will probably need to file a bug for it. so that's my second question :-) | 19:23 |
| clarkb | we don't really track bugs for individual projects unless they are causing the gate to crumble | 19:23 |
| *** mrmartin has joined #openstack-infra | 19:23 | |
| kevinbenton | clarkb: where should a bug be filed against something in the openstack common directory? | 19:23 |
| clarkb | kevinbenton: no that is a neutron bug | 19:23 |
| clarkb | looks like neutron isn't properly mocking out a dir | 19:24 |
| clarkb | (/var/lib/neutron) | 19:24 |
| *** dkranz has joined #openstack-infra | 19:25 | |
| kevinbenton | clarkb: oh, great. another race condition in mocking | 19:25 |
| kevinbenton | :-( | 19:25 |
| anteaya | kevinbenton: testr run --parallel --concurrency N --until-failure | 19:26 |
| anteaya | to flush out race conditions in unit tests | 19:27 |
| kevinbenton | anteaya, clarkb: thanks | 19:29 |
| *** dstanek has joined #openstack-infra | 19:30 | |
| *** jp_at_hp has quit IRC | 19:33 | |
| kevinbenton | anteaya, clarkb: what is the concurrency level of the jenkins nodes? | 19:36 |
| clarkb | kevinbenton: depends. but either 4 or 8 depending on provider/AZ | 19:36 |
| fungi | kevinbenton: in what sense. the jenkins executor count? | 19:36 |
| fungi | ahh, you mean in terms of testr | 19:37 |
| kevinbenton | fungi: no, parallel threads for testr. trying to figure out race problem | 19:37 |
| *** mrmartin has quit IRC | 19:37 | |
| clarkb | the test that failed used 4 http://logs.openstack.org/77/84377/2/gate/gate-neutron-python27/55e30bd/console.html#_2014-04-01_18_23_45_669 | 19:37 |
| kevinbenton | clarkb: thanks | 19:37 |
| *** zns has quit IRC | 19:38 | |
| *** jp_at_hp has joined #openstack-infra | 19:40 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Puppet nodes are unresponsive after 30 minutes https://review.openstack.org/84532 | 19:40 |
| nibalizer | jeblair: ^ | 19:40 |
| fungi | nibalizer: awesome--thanks again | 19:44 |
| nibalizer | :) | 19:44 |
| *** gokrokve has joined #openstack-infra | 19:46 | |
| *** hashar has joined #openstack-infra | 19:46 | |
| *** maxbit has quit IRC | 19:47 | |
| *** e0ne_ has joined #openstack-infra | 19:51 | |
| *** e0ne has quit IRC | 19:51 | |
| *** e0ne_ has quit IRC | 19:53 | |
| *** e0ne has joined #openstack-infra | 19:54 | |
| *** sdake_ is now known as sdake__ | 19:54 | |
| *** yamahata has joined #openstack-infra | 19:55 | |
| *** CaptTofu has quit IRC | 19:55 | |
| *** CaptTofu has joined #openstack-infra | 19:56 | |
| *** ociuhandu has quit IRC | 19:56 | |
| *** thuc has joined #openstack-infra | 19:56 | |
| *** CaptTofu has quit IRC | 20:00 | |
| clarkb | pretty sure that RBAC stuff is not openstack though | 20:00 |
| *** dkliban has joined #openstack-infra | 20:01 | |
| fungi | clarkb: so what we're passing into openstackwatch.ini is called "swift_store_key" in hiera, but appears to actually be the rax special api key for that tenant | 20:02 |
| mordred | notmyname: ping | 20:02 |
| fungi | i guess it wasn't a dedicated api key after all | 20:02 |
| notmyname | yo | 20:02 |
| clarkb | fungi: ah ok | 20:02 |
| *** zns has joined #openstack-infra | 20:02 | |
| clarkb | fungi: so I think we have to use !openstack to make this work | 20:02 |
| mordred | so - we're all asking each other swift questions - I figure you know the answer | 20:02 |
| clarkb | fungi: doable with that key or RBAC | 20:02 |
| clarkb | but less than ideal | 20:02 |
| mordred | notmyname: if we want to give a program the ability to write intoa swift container | 20:02 |
| notmyname | mordred: yes swift has container acls ;-) | 20:02 |
| clarkb | notmyname: right I don't think anyone is contesting that | 20:02 |
| *** markmcclain has joined #openstack-infra | 20:03 | |
| clarkb | its the application of them in particular clouds that is problematic | 20:03 |
| notmyname | mordred: that's going to be somewhat dependent on the auth system. eg RAX auth just recently supported it | 20:03 |
| clarkb | this ^ | 20:03 |
| mordred | ah. gotcha | 20:03 |
| *** mwagner_afk has quit IRC | 20:03 | |
| clarkb | anyways I need to grab lunch | 20:03 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 20:03 |
| mordred | cool. that helps. thanks notmyname | 20:03 |
| *** zns has quit IRC | 20:03 | |
| clarkb | jhesketh: jeblair ^ reviews on what I have done with the tests in 76057 would be greaty | 20:03 |
| notmyname | mordred: clarkb: in general, swift has an opaque bucket in which to store "acls", but they are interpreted by the particular auth system running | 20:04 |
| clarkb | jhesketh: jeblair I think that opens the door to doing more testing with non standard config | 20:04 |
| clarkb | so this will end up being another dns like situation. Not terrible but not great | 20:04 |
| clarkb | we can probably live iwth it | 20:04 |
| mordred | notmyname: well, we're REALLY close to starting to store stuff in swift - so there may be further follow up questions :) | 20:04 |
| notmyname | mordred: great! | 20:05 |
| *** derekh has quit IRC | 20:05 | |
| fungi | poking in the rax web dashboard, looks like each sub-user in a tenant gets assigned a dedicated api key. suspecting this means having to use the pyrax extension in the client? | 20:05 |
| jhesketh | clarkb: yeah, not sure how to bring that into tests for cloud providers besides just trying and seeing what works | 20:05 |
| fungi | oh, apparently they get a username and password too, so maybe normal client operations would support this | 20:07 |
| *** CaptTofu has joined #openstack-infra | 20:08 | |
| fungi | ahh, the access controls in the rax dashboard, at least, are limited to either granting the account access to all containers, or none. there doesn't seem to be per-container granularity there | 20:09 |
| fungi | so we could limit the account to only be able to read and write (or only be able to read) all storage containers on the tenant and have no access to create servers, cinder volumes, et cetera if we wanted | 20:10 |
| jeblair | we could also go with public access | 20:11 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Protect against /etc/nodepool not existing https://review.openstack.org/84492 | 20:11 |
| jeblair | i'm not sure the cdn issues we had are actually blockers (i think it mostly had to do with making new dns cnames for each container) | 20:11 |
| *** lcostantino has quit IRC | 20:11 | |
| jeblair | but if we don't need to cname them at all, then it's not a big deal | 20:12 |
| fungi | more just wondering how we effectively limit write access per container | 20:12 |
| jeblair | fungi: yeah, that would still be good | 20:13 |
| jeblair | it's just possible we might want a container not controlled by zuul. :) | 20:13 |
| *** yolanda has quit IRC | 20:13 | |
| fungi | if there's support in the api, it's at least not reflected in their webui | 20:13 |
| *** Sukhdev has quit IRC | 20:14 | |
| *** CaptTofu has quit IRC | 20:14 | |
| *** CaptTofu has joined #openstack-infra | 20:15 | |
| *** e0ne has quit IRC | 20:16 | |
| jeblair | fungi: i guess we may need to put it in openstackjenkins then | 20:17 |
| *** e0ne has joined #openstack-infra | 20:17 | |
| fungi | jeblair: for the reasons clarkb (who is apparently smarter than me) outlined earlier, i agree ;) | 20:18 |
| jeblair | fungi: yeah, it just never occured to me that with all the talk of container acls, etc, that they would not actually exist | 20:18 |
| mordred | fungi: I think one of these days we're going to discover taht clarkb has actually stolen all of our brains | 20:18 |
| fungi | mordred: i asked him to at least give me a receipt for mine | 20:18 |
| mordred | fungi: good luck with that | 20:19 |
| clarkb | :P | 20:19 |
| *** CaptTofu has quit IRC | 20:19 | |
| clarkb | fungi: one brain, slightly mushy received April first | 20:19 |
| fungi | danke | 20:19 |
| * fungi vaporizes for a few minutes to cook dinner before his next meeting | 20:20 | |
| jeblair | notmyname: do you know how to use container acls with rackspace? | 20:20 |
| notmyname | jeblair: I read the blog from last week. | 20:21 |
| jeblair | notmyname: ah, i see it. does swiftclient support setting those things that the blog post describes using curl for? | 20:23 |
| jeblair | http://www.rackspace.com/blog/create-cloud-files-container-level-access-control-policies/ | 20:23 |
| jeblair | i assume that's the blog post ^ | 20:23 |
| *** gyee has joined #openstack-infra | 20:24 | |
| *** julim has quit IRC | 20:25 | |
| notmyname | jeblair: I think it's in swiftclient. checking | 20:26 |
| *** mrda_away is now known as mrda | 20:27 | |
| *** branen has joined #openstack-infra | 20:28 | |
| notmyname | jeblair: -r for read ACLs and -w for write ACLs. (`swift post --help`) | 20:29 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Limit Tempest tests in virutal Ironic devstack job https://review.openstack.org/84543 | 20:29 |
| notmyname | jeblair: and yes, it looks like RAX is using the normal ACL headers, so everything should be fine | 20:29 |
| notmyname | jeblair: FWIW, I haven't actually tried it | 20:29 |
| jeblair | notmyname: cool, thanks much! i think we might soon. :) | 20:30 |
| jeblair | fungi, jhesketh, clarkb: ^ | 20:30 |
| *** zns has joined #openstack-infra | 20:30 | |
| *** jcoufal has quit IRC | 20:37 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: turn back on log enforcement https://review.openstack.org/81216 | 20:39 |
| *** ociuhandu has joined #openstack-infra | 20:40 | |
| jeblair | dkranz, mtreinish, sdague: ^ | 20:40 |
| sdague | woot | 20:40 |
| dkranz | jeblair: yes! | 20:40 |
| *** dizquierdo has joined #openstack-infra | 20:42 | |
| *** dims_ has quit IRC | 20:44 | |
| sdague | clarkb / fungi: https://review.openstack.org/#/c/84469/ should land that as well, as it was somewhat coupled with 81216 | 20:46 |
| *** CaptTofu has joined #openstack-infra | 20:47 | |
| *** mfer has joined #openstack-infra | 20:47 | |
| *** mspreitz has joined #openstack-infra | 20:47 | |
| mspreitz | I have a procedural question. If two developers want to collaborate on a patch before submitting it for review, what is the right way to do that? | 20:47 |
| *** CaptTofu has quit IRC | 20:48 | |
| *** CaptTofu_ has joined #openstack-infra | 20:48 | |
| clarkb | back from lunch | 20:48 |
| *** dkliban has quit IRC | 20:49 | |
| clarkb | mspreitz: you are free to do it however you like | 20:49 |
| anteaya | mspreitz: well one of them has to submit it to gerrit, so one will be considered the owner | 20:49 |
| clarkb | mspreitz: there is no right or wrong way, we have no enforcement on the space | 20:49 |
| *** CaptTofu_ has quit IRC | 20:49 | |
| anteaya | the owner can set wip and abandonment and reinstatement | 20:50 |
| mspreitz | So we can do it via Gerrit. | 20:50 |
| clarkb | mspreitz: that said there shouldn't be anything that prevents you using gerrit immediately and iterating on a change using WIP | 20:50 |
| mspreitz | This patch is for a blueprint that has not yet been submitted... so what do we use for a branch name in Gerrit? | 20:50 |
| anteaya | mspreitz: either of you can pull down the change and submit | 20:50 |
| anteaya | you can change the branch name later | 20:51 |
| *** thomasem_ has quit IRC | 20:51 | |
| mtreinish | jeblair: awesome | 20:51 |
| anteaya | future-bp or something | 20:51 |
| clarkb | branch names are arbitrary | 20:51 |
| mspreitz | thanks | 20:51 |
| *** CaptTofu_ has joined #openstack-infra | 20:53 | |
| clarkb | jeblair: jhesketh woot zuul was happy with https://review.openstack.org/#/c/76057 which makes me happy | 20:54 |
| *** blamar has quit IRC | 20:58 | |
| *** blamar has joined #openstack-infra | 21:00 | |
| *** david-lyle_afk is now known as david-lyle | 21:01 | |
| *** aysyd has quit IRC | 21:01 | |
| *** rfolco has quit IRC | 21:02 | |
| *** cyeoh has quit IRC | 21:04 | |
| *** blamar has quit IRC | 21:04 | |
| *** jeckersb is now known as jeckersb_gone | 21:04 | |
| *** cyeoh has joined #openstack-infra | 21:05 | |
| *** blamar has joined #openstack-infra | 21:06 | |
| *** CaptTofu_ has quit IRC | 21:07 | |
| clarkb | fungi: jeblair: are you guys ready for me to put nodepool back into puppeting? | 21:08 |
| *** CaptTofu has joined #openstack-infra | 21:08 | |
| clarkb | fungi: jeblair: my plan for that is checkout master in /opt/nodepool, start puppet, then at some point we should restart nodepool | 21:08 |
| *** mfer has quit IRC | 21:08 | |
| clarkb | (puppet should install nodepool using the fix for /etc/nodepool not existing) | 21:08 |
| jeblair | clarkb: ++ | 21:11 |
| *** CaptTofu has quit IRC | 21:12 | |
| mattoliverau | Morning | 21:12 |
| *** blamar has quit IRC | 21:13 | |
| *** blamar has joined #openstack-infra | 21:14 | |
| *** dprince has quit IRC | 21:14 | |
| *** pdmars has quit IRC | 21:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: turn off clean log enforcement in stress tests https://review.openstack.org/84469 | 21:15 |
| *** pcrews_ has quit IRC | 21:15 | |
| *** alff_ has quit IRC | 21:17 | |
| *** alff has quit IRC | 21:17 | |
| *** blamar has quit IRC | 21:18 | |
| *** blamar has joined #openstack-infra | 21:18 | |
| *** pcrews_ has joined #openstack-infra | 21:18 | |
| *** dkliban has joined #openstack-infra | 21:18 | |
| *** e0ne has quit IRC | 21:18 | |
| *** jp_at_hp has quit IRC | 21:19 | |
| jogo | http://bit.ly/1lAFeUb | 21:20 |
| jogo | looks like new neutron unit test failure | 21:20 |
| clarkb | jogo: yup | 21:20 |
| jogo | my guess is out of memory or something | 21:20 |
| clarkb | kevinbenton was looing at it before | 21:20 |
| jogo | err disk space | 21:20 |
| *** blamar_ has joined #openstack-infra | 21:20 | |
| clarkb | jogo: no, that looks like tests that should be using tmpdir fixtures that don't | 21:20 |
| clarkb | jogo: jenkins user can't write to /var/lib/neutron | 21:21 |
| *** mspreitz has quit IRC | 21:21 | |
| kevinbenton | clarkb: still looking at it. it doesn't make any sense | 21:21 |
| jogo | salv-orlando: ^ ahh | 21:21 |
| fungi | clarkb: sounds safe enough | 21:21 |
| jogo | clarkb: thanks, salv-orlando is preping an e-r patch for this bug | 21:21 |
| salv-orlando | kevinbenton: do you have a bug number? | 21:22 |
| kevinbenton | salv-orlando: no because i haven't determined the root cause yet. if i clone master and then checkout an old commit. it still fails | 21:22 |
| kevinbenton | salv-orlando: might be a requirements issue | 21:23 |
| clarkb | kevinbenton: salv-orlando: I would try running testr run --analyze isolation against it | 21:23 |
| clarkb | which should point out conflicting tests if that is causing it | 21:23 |
| *** blamar has quit IRC | 21:23 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Test stable branch on client changes https://review.openstack.org/80698 | 21:24 |
| kevinbenton | clarkb: not necessary. just run tox -v -epy27 neutron.tests.unit.test_l3_agent.TestL3AgentEventHandler.test_spawn_metadata_proxy | 21:25 |
| kevinbenton | clarkb: that's one of the failing tests. just fails on it's own | 21:25 |
| *** blamar has joined #openstack-infra | 21:25 | |
| kevinbenton | clarkb: that's why it seems like something externally changesd | 21:25 |
| *** blamar_ has quit IRC | 21:25 | |
| kevinbenton | changed* | 21:25 |
| salv-orlando | I'll then file a generic bug in neutron just for tracking the issue with an e-r query | 21:25 |
| kevinbenton | salv-orlando: ok | 21:26 |
| clarkb | kevinbenton: oh interesting | 21:26 |
| salv-orlando | kevinbenton: if you're looking at requirements, markmcclain pointed this patch | 21:26 |
| salv-orlando | https://review.openstack.org/#/c/83474/ | 21:26 |
| kevinbenton | salv-orlando: so the strange part is that it's still broken if i do a fresh clone and checkout a commit before that | 21:29 |
| *** adalbas has quit IRC | 21:29 | |
| kevinbenton | salv-orlando: e.g. 3cc5b4d0a312995d3bf5e4314c8a80df73ebd3e1 | 21:29 |
| clarkb | fungi: jeblair: proceding with nodepool puppet corrective action now | 21:29 |
| jogo | salv-orlando: link to e-r patch? | 21:29 |
| salv-orlando | cool, do you run tox -r | 21:29 |
| clarkb | I am not going to restart nodepool unless we have more than one person around to help out :) learned my lesson yesterday | 21:29 |
| jogo | salv-orlando: is this a 100% failure? | 21:29 |
| fungi | clarkb: thanks | 21:30 |
| jeblair | clarkb: am around | 21:30 |
| kevinbenton | salv-orlando: no, what does that do? | 21:30 |
| jogo | if so we can just try doign a revert and see what happens | 21:30 |
| * fungi is around again too for the next half hour at least | 21:30 | |
| salv-orlando | tox -r will download again all the dependencies | 21:30 |
| *** wenlock_ has joined #openstack-infra | 21:30 | |
| salv-orlando | jogo: does not seem 100%. Finishing up e-r query and then I'll fetch exact numbers | 21:30 |
| kevinbenton | salv-orlando: oh, yeah it's still broken in that case | 21:30 |
| kevinbenton | salv-orlando: just tried | 21:31 |
| kevinbenton | salv-orlando: i was barbarically deleting the .tox and .testrepository directories before :-) | 21:31 |
| *** dizquierdo has quit IRC | 21:31 | |
| kevinbenton | so i suspect a version of something bumped upwards and broke backwards compatibility | 21:31 |
| kevinbenton | one of the >= requirements | 21:32 |
| *** markmc has quit IRC | 21:32 | |
| clarkb | jeblair: fungi: in that case I am triggering a service restart in a root screen on nodepool.o.o shortly | 21:32 |
| clarkb | so that it will gracefully restart | 21:33 |
| kevinbenton | someone must have released an april fools package | 21:33 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a fingerprint for bug 1301027 https://review.openstack.org/84558 | 21:34 |
| uvirtbot | Launchpad bug 1301027 in neutron "Permission denied error in unit tests" [Undecided,New] https://launchpad.net/bugs/1301027 | 21:34 |
| *** dkranz has quit IRC | 21:34 | |
| clarkb | jeblair: fungi: restart is being triggered now | 21:34 |
| *** wenlock_ has quit IRC | 21:35 | |
| clarkb | wow that was quick | 21:35 |
| clarkb | it is running the new stuff now | 21:35 |
| salv-orlando | jogo0: failure rate is at 100% | 21:35 |
| fungi | i have, on occasion, had surprisingly but pleasantly quick graceful restarts from nodepool | 21:35 |
| jogo | salv-orlando: so 100% is good at least its easy to reproduce ;) | 21:36 |
| salv-orlando | fwiw e-r query is here: https://review.openstack.org/#/c/84558/ | 21:36 |
| salv-orlando | but with a 100% failure rate we don't need it | 21:36 |
| jogo | salv-orlando: yeah will holdoff on +2ing that until we have a fix .. so we can get better unclassified failure numbers | 21:37 |
| clarkb | seeing OverLimit: Quota exceeded for ram: Requested 16384, but already used 952320 of 960000 ram (HTTP 413) (Request-ID: req-f6655aec-2e77-4e52-81b4-1db50ede65ee) from tripleo again | 21:38 |
| fungi | clarkb: so i guess next we might want to test triggering an image update. i can take that on | 21:38 |
| clarkb | fungi: go for it and ++ | 21:38 |
| fungi | should i pick something relatively unused like new kernel or py3k? | 21:39 |
| clarkb | fungi: sure? | 21:39 |
| fungi | though i guess if it succeeds in building an image, we're most of the way there anyway | 21:39 |
| jeblair | fungi: if something fails (like mkdir /etc/nodepool), it should abort, so i wouldn't worry too much | 21:40 |
| salv-orlando | kevinbenton: if you're tracking this please assign bug 1301027 to yourself. I won't be around for long. | 21:40 |
| uvirtbot | Launchpad bug 1301027 in neutron "Permission denied error in unit tests" [Undecided,New] https://launchpad.net/bugs/1301027 | 21:40 |
| jeblair | clarkb: there are no aliens and tripleo has < 55 nodes | 21:41 |
| *** thuc has quit IRC | 21:41 | |
| jeblair | clarkb: so the math doesn't add up | 21:41 |
| kevinbenton | salv-orlando: will do | 21:41 |
| jeblair | clarkb: taking to tripleo | 21:41 |
| clarkb | jeblair: yup, I think that either the bug wasn't fixed or the explosions broke it in new ways | 21:41 |
| salv-orlando | kevinbenton: thanks | 21:41 |
| *** thuc has joined #openstack-infra | 21:41 | |
| jogo | so the timing for oslo.config fits | 21:42 |
| kevinbenton | version bump in olso | 21:43 |
| kevinbenton | jogo: yep | 21:43 |
| lifeless | mikal: around ? | 21:43 |
| salv-orlando | jogo: oslo.config change might also explain how we can have 100% failure rate in the gate without having seen failures in check queue first | 21:43 |
| kevinbenton | jogo: testing pinned to a lower version now | 21:43 |
| *** mriedem has quit IRC | 21:44 | |
| clarkb | I am seeing ready nodes that look very new, I think nodepool is happy | 21:45 |
| clarkb | will continue to monitor it | 21:45 |
| *** thuc has quit IRC | 21:46 | |
| *** hashar has quit IRC | 21:46 | |
| kevinbenton | salv-orlando, clarkb, jogo: lower version fixed it. //review.openstack.org/84562 | 21:47 |
| kevinbenton | http://review.openstack.org/84562 | 21:47 |
| jogo | dhellmann: ^ | 21:47 |
| clarkb | kevinbenton: will that pass tests? | 21:47 |
| clarkb | https://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n48 may need to be updated first | 21:48 |
| kevinbenton | clarkb: i didn't run the full suite. just the broken ones | 21:48 |
| clarkb | or, is it possible neutron is just using oslo.config in a weird way | 21:48 |
| fungi | mmm, my image-update died in a paramiko eoferror... so much fun. restarting | 21:48 |
| kevinbenton | i think it could be something related to the evaluation of '$state' in the unit tests | 21:49 |
| jogo | hopefully dhellmann is still around to help on that side | 21:49 |
| clarkb | jogo: I think he mentioned in another channel that his battery was dying | 21:49 |
| *** salv-orlando_ has joined #openstack-infra | 21:49 | |
| *** enikanorov has joined #openstack-infra | 21:49 | |
| kevinbenton | like now it's being evaluated on definition rather than reference | 21:49 |
| jogo | clarkb: oh yeah | 21:49 |
| *** e0ne has joined #openstack-infra | 21:49 | |
| fungi | i didn't even realize dhellmann was a robot, but that makes sense now | 21:50 |
| jogo | whats the oslo room name, lets go over there and ask em | 21:50 |
| *** pcrews_ has quit IRC | 21:50 | |
| *** erw_ has joined #openstack-infra | 21:51 | |
| jogo | kevinbenton: confirmed your patch works wonder why ... | 21:51 |
| *** CaptTofu has joined #openstack-infra | 21:51 | |
| *** e0ne_ has joined #openstack-infra | 21:51 | |
| *** e0ne has quit IRC | 21:51 | |
| salv-orlando_ | oslo.config works in mysterious ways | 21:51 |
| kevinbenton | jogo: i was suspecting something changed in the cfg parsing of that $state var | 21:52 |
| *** dstanek has quit IRC | 21:52 | |
| jogo | * Fix for parsing error with Dollar Sign ($) in values | 21:52 |
| kevinbenton | that sounds relevant | 21:52 |
| kevinbenton | jogo: https://github.com/bigswitch/neutron/blob/master/neutron/agent/linux/external_process.py#L31 | 21:52 |
| jogo | ahhh | 21:53 |
| * jogo pulls up the oslo patch | 21:53 | |
| *** dguerri_ has joined #openstack-infra | 21:53 | |
| *** sdake_ has joined #openstack-infra | 21:53 | |
| *** sdake_ has quit IRC | 21:53 | |
| *** sdake_ has joined #openstack-infra | 21:53 | |
| *** mtaylor has joined #openstack-infra | 21:53 | |
| *** jpeeler1 has joined #openstack-infra | 21:53 | |
| *** sdake_ is now known as sdake_1 | 21:53 | |
| clarkb | so state_path should probably be a fixtures.TmpDir() or whatever that class is | 21:53 |
| jogo | kevinbenton: I4324bda6f3de3cdf05f8eb3cf16052c04018dec8 | 21:53 |
| jogo | clarkb: yeah | 21:54 |
| clarkb | and be set for every test rather than using a global value | 21:54 |
| jogo | and this needs to be applied to the icehouse branch too | 21:54 |
| *** signed8b_ has joined #openstack-infra | 21:54 | |
| *** DinaBelova2 has joined #openstack-infra | 21:54 | |
| *** signed8bit has quit IRC | 21:54 | |
| *** gaelL_ has quit IRC | 21:54 | |
| *** dguerri has quit IRC | 21:54 | |
| *** jpeeler has quit IRC | 21:54 | |
| *** ruhe has quit IRC | 21:54 | |
| *** openstackgerrit has quit IRC | 21:54 | |
| *** salv-orlando has quit IRC | 21:54 | |
| *** DinaBelova has quit IRC | 21:54 | |
| *** mordred has quit IRC | 21:54 | |
| *** erw has quit IRC | 21:54 | |
| *** sdake has quit IRC | 21:54 | |
| *** Alexandra_ has quit IRC | 21:54 | |
| *** enikanorov_ has quit IRC | 21:54 | |
| *** signed8b_ has quit IRC | 21:54 | |
| *** signed8b_ has joined #openstack-infra | 21:54 | |
| *** salv-orlando_ is now known as salv-orlando | 21:54 | |
| *** gaelL has joined #openstack-infra | 21:54 | |
| *** erw_ is now known as erw | 21:54 | |
| clarkb | jogo: possibly, yes | 21:54 |
| *** DinaBelova2 is now known as DinaBelova | 21:54 | |
| *** ruhe has joined #openstack-infra | 21:55 | |
| *** Alexandra_ has joined #openstack-infra | 21:55 | |
| mikal | lifeless: wassup, was in meetings | 21:55 |
| *** e0ne_ has quit IRC | 21:56 | |
| lifeless | mikal: hi; you asked for some data on the quota skew thing | 21:56 |
| lifeless | mikal: I'm not sure what data you wanted | 21:56 |
| lifeless | mikal: I don't want to leave the quota wrong, but I don't want to destroy evidence | 21:56 |
| kevinbenton | clarkb: that will be changes in several places. is it necessary for oslo to immediately substitute? | 21:57 |
| mikal | lifeless: Hmmm, I don't recall the bug number. Have you got it handy? | 21:57 |
| *** mfer has joined #openstack-infra | 21:57 | |
| *** mfer has quit IRC | 21:57 | |
| lifeless | 1284424 | 21:58 |
| lifeless | mikal: ^ | 22:00 |
| mikal | Ta | 22:00 |
| *** dstanek has joined #openstack-infra | 22:01 | |
| *** zehicle has joined #openstack-infra | 22:02 | |
| *** weshay has quit IRC | 22:04 | |
| *** gothicmindfood has left #openstack-infra | 22:05 | |
| mikal | lifeless: so let me see if I am getting this straight... | 22:05 |
| *** zehicle_at_dell has quit IRC | 22:05 | |
| mikal | lifeless: the quotas are wrong because some instances are being double counted? | 22:05 |
| clarkb | kevinbenton: I can't speak for oslo but that is a test bug imo | 22:05 |
| *** dims_ has joined #openstack-infra | 22:06 | |
| clarkb | kevinbenton: even if you weren't using oslo you would want your throwaway data dirs to be managed by a proper fixture | 22:06 |
| lifeless | mikal: we don't know the cause | 22:06 |
| lifeless | mikal: but we know it gets very very out | 22:06 |
| lifeless | mikal: e.g. jeblair says its out right now | 22:07 |
| *** pcrews has joined #openstack-infra | 22:07 | |
| mikal | lifeless: but out in the direction of over counting, yes? | 22:07 |
| mikal | lifeless: what sort of outage caused the problem this time? Last time it was a control plane outage, yes? | 22:07 |
| lifeless | mikal: the user has more 'used' than the sum of their vms. | 22:07 |
| anteaya | four programs now in need of at least one candidate: https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014#Candidates | 22:08 |
| lifeless | mikal: yes, this time the control plane has been robust but the hypervisors have been falling over like dead flies | 22:08 |
| anteaya | hint hint hint | 22:08 |
| *** denis_makogon has quit IRC | 22:08 | |
| *** dcramer_ has quit IRC | 22:08 | |
| mikal | lifeless: hmmm, I need to think on this | 22:08 |
| mikal | lifeless: I agree its a bug, I just need to work out where to start poking | 22:09 |
| *** gothicmindfood has joined #openstack-infra | 22:09 | |
| lifeless | mikal: is there anything we can do to make poking possible before we destroy the evidence? | 22:09 |
| *** gothicmindfood has quit IRC | 22:09 | |
| lifeless | mikal: since running less slaves than the hardware can is undesirable | 22:09 |
| mikal | lifeless: I'd say do a dump of the quota table, and email it to me | 22:09 |
| mikal | Then destroy with vigor | 22:09 |
| kevinbenton | clarkb: right | 22:10 |
| lifeless | SpamapS: ^ thats something you can do to help | 22:10 |
| * mikal is thinking that perhaps punching devstack nova-compute in the face over and over might duplicate this problem | 22:10 | |
| lifeless | SpamapS: get that dump, send to mikal, reset the quota | 22:10 |
| *** gothicmindfood has joined #openstack-infra | 22:10 | |
| fungi | INFO:nodepool.ImageUpdater:Image dsvm-precise-krnl-1396388920.template.openstack.org in rax-dfw is ready | 22:10 |
| fungi | now i'll delete all ready nodes for that type | 22:10 |
| *** hashar has joined #openstack-infra | 22:11 | |
| SpamapS | ok | 22:11 |
| SpamapS | mikal: just quota_usages ? | 22:11 |
| mikal | SpamapS: and the instances table as well perhaps? | 22:12 |
| mikal | Not sure how big that is in your world | 22:12 |
| lifeless | 24K rows | 22:13 |
| mikal | Small | 22:13 |
| *** signed8b_ has quit IRC | 22:13 | |
| mikal | Dump it as well then pleace | 22:13 |
| lifeless | new cloud | 22:13 |
| lifeless | only been running 48 hours | 22:13 |
| mikal | Or if you're super keen, mysqldump the world | 22:13 |
| mikal | That would end the discussion of what tables | 22:13 |
| *** dstanek has quit IRC | 22:14 | |
| lifeless | and boom things are connected | 22:14 |
| SpamapS | ugh | 22:14 |
| *** alff has joined #openstack-infra | 22:14 | |
| *** alff_ has joined #openstack-infra | 22:14 | |
| SpamapS | our new percona static binary overlords do not install mysqldump in the path | 22:14 |
| *** alexpilotti has quit IRC | 22:14 | |
| fungi | SpamapS: you know you love them anyway | 22:15 |
| lifeless | 47 slaves running / building | 22:15 |
| lifeless | that seems good | 22:15 |
| lifeless | jeblair: ^ | 22:15 |
| lifeless | broker is back up | 22:15 |
| lifeless | we should see things happening any second now | 22:15 |
| fungi | dsvm-precise-krnl-rax-dfw-3380665.slave.openstack.org is building | 22:15 |
| fungi | once it's up, i'll make sure it has the bits we expect in /etc/nodepool | 22:16 |
| fungi | or whether they ended up in /etc/nodepolo instead | 22:16 |
| clarkb | fungi: marco! | 22:16 |
| fungi | (...ni) | 22:17 |
| fungi | which apparently still needs a ptl | 22:17 |
| *** gothicmindfood has quit IRC | 22:17 | |
| SpamapS | lifeless: dump made, usages reset | 22:17 |
| fungi | clarkb: you should run for marconi ptl and spearhead the rename | 22:17 |
| lifeless | unused floating-ips deleting now | 22:18 |
| *** dims_ has quit IRC | 22:18 | |
| clarkb | fungi: ha | 22:18 |
| *** signed8bit has joined #openstack-infra | 22:19 | |
| fungi | run on an all-pasta platform. it would capture 100% of the electorate, no doubt | 22:19 |
| *** bhuvan has joined #openstack-infra | 22:20 | |
| jogo | fungi: not devanand | 22:20 |
| jogo | devananda: | 22:20 |
| clarkb | you can build a computer with devanand gates | 22:21 |
| *** gothicmindfood has joined #openstack-infra | 22:21 | |
| fungi | devanxor gates wouldn't work nearly as well | 22:21 |
| lifeless | fungi: the great flying spaghetti monster for ever? | 22:21 |
| jeblair | clarkb: sNORt! | 22:21 |
| fungi | lifeless: may his noodly appendage touch you inappropriately | 22:21 |
| clarkb | ok back to code review I go :) | 22:22 |
| lifeless | ok this https://jenkins04.openstack.org/job/check-tripleo-ironic-seed-precise/108/ is on a new slave post-broker bring-up | 22:22 |
| lifeless | if its all good it will get a testenv and be away pronto | 22:22 |
| *** jgrimm has quit IRC | 22:23 | |
| *** bhuvan has quit IRC | 22:24 | |
| *** jaypipes has joined #openstack-infra | 22:24 | |
| *** bhuvan has joined #openstack-infra | 22:24 | |
| *** thuc has joined #openstack-infra | 22:25 | |
| SpamapS | mikal: dump sent | 22:25 |
| *** thuc has quit IRC | 22:25 | |
| mikal | Ta | 22:26 |
| *** thuc has joined #openstack-infra | 22:26 | |
| *** signed8bit has quit IRC | 22:28 | |
| *** signed8bit has joined #openstack-infra | 22:30 | |
| *** changbl has quit IRC | 22:32 | |
| lifeless | 37 test environments registered | 22:32 |
| lifeless | hmm, we probably need to sacrifice a couple hypervisors for more TE hosts | 22:33 |
| *** mspreitz has joined #openstack-infra | 22:33 | |
| lifeless | though, I suspect the outage may have taken a couple tes down as 37 is a strange number to get from 10 hosts | 22:33 |
| lifeless | or perhaps that was the neutron quota limit being hit | 22:33 |
| *** esker has quit IRC | 22:34 | |
| *** CaptTofu has quit IRC | 22:34 | |
| *** CaptTofu has joined #openstack-infra | 22:35 | |
| *** eharney has quit IRC | 22:35 | |
| *** dims_ has joined #openstack-infra | 22:36 | |
| clarkb | jeblair: https://review.openstack.org/#/c/81320/ lgtm | 22:36 |
| *** khyati has quit IRC | 22:36 | |
| *** BobBall_ has joined #openstack-infra | 22:38 | |
| *** BobBall has quit IRC | 22:38 | |
| *** prad has quit IRC | 22:38 | |
| *** UtahDave has quit IRC | 22:39 | |
| *** CaptTofu has quit IRC | 22:39 | |
| *** nkinder has quit IRC | 22:40 | |
| *** dstanek has joined #openstack-infra | 22:40 | |
| *** BobBall_ has quit IRC | 22:41 | |
| *** thuc has quit IRC | 22:42 | |
| *** thuc has joined #openstack-infra | 22:42 | |
| *** khyati has joined #openstack-infra | 22:42 | |
| *** thedodd has quit IRC | 22:43 | |
| *** BobBall has joined #openstack-infra | 22:43 | |
| *** BobBall is now known as Guest86798 | 22:44 | |
| lifeless | hmm none of these jobs seem to be acquiring a testenv succesfully | 22:46 |
| *** thuc has quit IRC | 22:46 | |
| lifeless | I might tear down the te's and bring them back up | 22:47 |
| *** jepoy has joined #openstack-infra | 22:48 | |
| *** rcleere has quit IRC | 22:49 | |
| asselin | anyone seen this error when using nodepool on ubuntu 12.04 image? http://paste.openstack.org/show/74781/ | 22:50 |
| *** markmcclain has quit IRC | 22:52 | |
| *** e0ne has joined #openstack-infra | 22:52 | |
| clarkb | zigo: are less css files a problem for debian packaging? | 22:54 |
| clarkb | zigo: I seem to remember that was a thing | 22:54 |
| *** khyati has quit IRC | 22:56 | |
| clarkb | zigo: nevermind found your response to the most recent less css thread | 22:57 |
| *** e0ne has quit IRC | 22:57 | |
| *** thuc has joined #openstack-infra | 22:57 | |
| *** thuc has quit IRC | 22:58 | |
| *** thuc has joined #openstack-infra | 22:59 | |
| *** saschpe has quit IRC | 23:00 | |
| fungi | clarkb: jeblair: lgtm... http://paste.openstack.org/show/74784/ | 23:02 |
| fungi | asselin: "you have held broken packages" | 23:03 |
| fungi | asselin: any idea why some packages on your base image might have a dpkg held status? | 23:03 |
| asselin | fungi, yes...but I can't figure out which ones. I'm using the devstack config scripts | 23:04 |
| fungi | er, hold status | 23:04 |
| *** michchap has quit IRC | 23:05 | |
| fungi | asselin: dpkg --get-selections|grep hold | 23:05 |
| fungi | asselin: that should list any held packages | 23:06 |
| asselin | nothing at all. I did see this: puppet-common : Depends: facter (< 2.0) but 2.0.1-1puppetlabs1 is to be installed | 23:06 |
| asselin | Recommends: debconf-utils but it is not going to be installed | 23:06 |
| asselin | perhaps something with a dependency. puppet depends on puppet-common | 23:06 |
| asselin | which ubuntu image does infra use? | 23:07 |
| fungi | asselin: 12.04 lts (precise pangolin) | 23:07 |
| *** yamahata has quit IRC | 23:08 | |
| asselin | fungi, that's what I'm using | 23:08 |
| asselin | precise-server-cloudimg-amd64-disk1.img | 23:08 |
| *** wenlock has quit IRC | 23:09 | |
| *** mriedem has joined #openstack-infra | 23:09 | |
| *** saschpe has joined #openstack-infra | 23:10 | |
| fungi | asselin: are you using something like http://git.openstack.org/cgit/openstack-infra/config/tree/install_puppet.sh to get puppet installed? (we call that script from prepare_node.sh) | 23:10 |
| asselin | fungi, this is suspicious no? puppet-common : Depends: facter (< 2.0) but 2.0.1-1puppetlabs1 is to be installed | 23:10 |
| asselin | yes...exactly that script | 23:10 |
| clarkb | fungi: jeblair: jhesketh I wasn't too harsh to https://review.openstack.org/#/c/82307/ was I | 23:11 |
| clarkb | no mordred here right now... | 23:12 |
| *** gokrokve has quit IRC | 23:12 | |
| jhesketh | clarkb: It's fair, but I've tidied it up a little down the path: https://review.openstack.org/#/c/84069/ | 23:13 |
| jhesketh | clarkb: I'd prefer to see it merged and then address the packaging issue so it doesn't block the rest of my changes, but clearly I'm biased | 23:14 |
| *** aarongr_ has joined #openstack-infra | 23:14 | |
| *** mspreitz has quit IRC | 23:14 | |
| asselin | fungi, FYI full log is here: http://paste.openstack.org/show/74785/ | 23:14 |
| clarkb | jhesketh: for the packaging thing I don't see any reason to switch to less | 23:16 |
| clarkb | the current css is dead simple (even I understand it) | 23:16 |
| clarkb | fungi: re Bug 1300843 maybe we can convince people to use real editors that put newlines at the end of files | 23:18 |
| uvirtbot | Launchpad bug 1300843 in openstack-ci "project-requirements-change gates master" [Medium,Triaged] https://launchpad.net/bugs/1300843 | 23:18 |
| clarkb | >_> | 23:18 |
| mattoliverau | Hey all, while building up Turbo-hipster nodes via nodepool, in my own environment, I use config/install_puppet.sh. As of this morning it is failing due to the pinned version of puppet-common needing to install a facter version of < 2.0.. so I think we will need to add anther pin. If I pin factor to version 1.* it workd s for me. | 23:18 |
| clarkb | mattoliverau: you may have solved asselin's problem | 23:19 |
| clarkb | mattoliverau: if that is the fix that works for me | 23:19 |
| *** marktraceur has quit IRC | 23:19 | |
| *** david-lyle has quit IRC | 23:20 | |
| fungi | asselin: all i can figure is that it's a difference in the base image. we run this (so far) on base images provided by rackspace and hpcloud... maybe wherever your base image came from, it had puppet components pre-installed? | 23:20 |
| mattoliverau | clarkb: well your very welcome :P | 23:20 |
| clarkb | fungi: see mattoliverau's comment | 23:20 |
| jhesketh | clarkb: so there's no strong reason to move to less, but with grunt building the system it's pretty easy.. it'll also let us retheme for infra using variables rather than overriding css | 23:21 |
| *** michchap has joined #openstack-infra | 23:21 | |
| *** _afazekas has quit IRC | 23:21 | |
| asselin | fungi, I got my image from ubuntu web-site. | 23:22 |
| clarkb | jhesketh: right its easy until someone tries to package it | 23:22 |
| clarkb | :) | 23:22 |
| asselin | clarkb, mattoliverau how did you result it? | 23:22 |
| mattoliverau | fungi: the puppetlabs repo I asusme have been updated, I started getting the same problem as asselin. I modified add a new pin and trying to install puppet again and it worked. | 23:23 |
| *** derekh has joined #openstack-infra | 23:23 | |
| jhesketh | clarkb: sure, I don't really know enough about it, maybe we should get some other opinions | 23:23 |
| jhesketh | nevertheless, should we merge it and if we end up not using less we can do that in a future patch? | 23:23 |
| clarkb | jhesketh: many people have chimed in re less and horizon in the past http://lists.openstack.org/pipermail/openstack-dev/2013-December/021584.html | 23:25 |
| clarkb | jhesketh: mostly relying on what zigo has said re debian packaging | 23:25 |
| fungi | clarkb: asselin: mattoliverau: ahh, yes! i'm multi-tasking too many discussions again | 23:25 |
| mattoliverau | lol | 23:25 |
| clarkb | jhesketh: http://lists.openstack.org/pipermail/openstack-dev/2013-December/021791.html is the specific response | 23:26 |
| fungi | mattoliverau: i wonder why the nodepool image-update i just did (using a precise base image in rackspace) didn't run into that issue | 23:26 |
| jeblair | jhesketh: i like using css for its intended purpose.. | 23:27 |
| fungi | unless this has cropped up just in the last 1.5 hours | 23:27 |
| jhesketh | I don't feel strongly about this (except that I wasted a lot of time yesterday making the build stuff work) | 23:28 |
| *** mspreitz has joined #openstack-infra | 23:29 | |
| jeblair | jhesketh: i haven't reviewed that patch, and i think it's going to be difficult for me to review because i don't understand any of those tools. | 23:29 |
| mattoliverau | fungi: I think it has. Cause earlier this morning it was working | 23:29 |
| clarkb | jeblair: that was partially my problem | 23:29 |
| mattoliverau | I've been playing with nodepool images for the last 2.5 hours. | 23:29 |
| fungi | mattoliverau: okay, then that explains my confusion nicely | 23:29 |
| clarkb | jeblair: there are literally 10 new files that are config or apply config | 23:29 |
| lifeless | ok we're back | 23:30 |
| clarkb | jeblair: to run 2 tests | 23:30 |
| lifeless | https://jenkins07.openstack.org/job/check-tripleo-undercloud-precise/373/console | 23:30 |
| lifeless | is running | 23:30 |
| *** thuc has quit IRC | 23:30 | |
| jeblair | jhesketh: my opinion on the matter is i think it should be a simple script that uses some css to say things like "this is a change; this is a job" | 23:30 |
| jhesketh | jeblair, clarkb: that's fair... perhaps you could pull in the tail of my changes and see that it works for you | 23:30 |
| jeblair | i think the last time we talked, we were all on board with that much at least | 23:30 |
| *** mrodden has quit IRC | 23:30 | |
| jeblair | i get the desire not to have that limit us in the future | 23:31 |
| jhesketh | jeblair: as in you don't want to use all of these build utils? | 23:31 |
| jeblair | a possible future where this is the start of a real app | 23:31 |
| jeblair | clearly that's not going to cut it at that point | 23:31 |
| jeblair | but i'm confused as to why it's necessary now -- does it help get us to the point where we want to be right now? | 23:32 |
| *** mwagner_afk has joined #openstack-infra | 23:32 | |
| jeblair | (which is that we stop having two versions of the status page?) | 23:32 |
| jhesketh | it helps with jshint and ensuring the code is clean | 23:32 |
| jhesketh | but you're right, it's not necessary | 23:32 |
| asselin | mattoliverau, clarkb how did you change the pin factor? I'd like to try it out | 23:33 |
| clarkb | jhesketh: I would be on board with the build tools if there was more documentation | 23:34 |
| mattoliverau | asselin: take a look in /etc/apt/preferences.d/ | 23:34 |
| *** marktraceur has joined #openstack-infra | 23:35 | |
| mgagne | With pbr, is there a way to derive the version from unsigned tags instead of GPG-signed tag? | 23:35 |
| clarkb | jhesketh: as is there is very little of what is going on there explained and several scripts are not used at all | 23:35 |
| clarkb | mgagne: I think pbr requires signed tags | 23:35 |
| mattoliverau | asselin: if you used the install_puppet.sh script there will be a 00-puppet.pref file | 23:35 |
| asselin | mattoliverau, thanks I see the file | 23:35 |
| mattoliverau | I added a new pin and the end of that file: | 23:35 |
| clarkb | mattoliverau: want to propose the change to gerrit? | 23:36 |
| mattoliverau | Package: facter | 23:36 |
| mattoliverau | Pin: version 1.* | 23:36 |
| mattoliverau | Pin-Priority: 501 | 23:36 |
| mattoliverau | clarkb: I sure can, I was just waiting on fungi to see if he can confirm it.. but I'll do it now anyway :) | 23:37 |
| *** nkinder has joined #openstack-infra | 23:37 | |
| mgagne | clarkb: bummer, signed tags are an esoteric concept to most devs here, I was hoping to have a way to smoothly introduce them to pbr without shocking them | 23:37 |
| jhesketh | jeblair: what's your feel re clarkb's request for more docs? | 23:37 |
| *** lcheng has quit IRC | 23:37 | |
| mattoliverau | asselin: ^^ | 23:37 |
| clarkb | mgagne: let me double check | 23:37 |
| fungi | mattoliverau: yeah, that seems like a good solution | 23:38 |
| fungi | mattoliverau: based on what i'm hearing, i expect if i try another image-update i'll see the same error you and asselin are | 23:39 |
| asselin | mattoliverau, that worked for me. | 23:39 |
| fungi | mgagne: that's too bad... cryptographically attesting to a release is... important | 23:39 |
| fungi | mgagne: especially to packagers and many end users | 23:40 |
| jeblair | jhesketh: i'm trying to be really honest here... i worry that the problem is that this change is about adding ~5 (i'm sure i'm not counting right) new tools that i have no experience with or understanding of their purpose or use | 23:41 |
| fungi | mgagne: anyway, i expect it's a baked-in assumption for pbr which would require a patch to change | 23:41 |
| mgagne | fungi: yha, I know. We can argue that it is only used for internal consumptions. Others might add that it doesn't matter and attesting releases of internal tools is as important as public ones. | 23:41 |
| jeblair | jhesketh: and i worry that writing docs about that in order to teach me about those probably isn't the right thing to do | 23:42 |
| jeblair | jhesketh: and also hypothetically if i understood all of them, would i think we got a benefit from them worth the cost of using them | 23:43 |
| clarkb | mgagne: ya, git describe --always doesn't seem to grok the unsigned tags | 23:43 |
| clarkb | mgagne: and that is what pbr is doing | 23:43 |
| mattoliverau | clarkb: http://is.gd/NMrFhP | 23:43 |
| clarkb | mattoliverau: awesome thnaks | 23:43 |
| mattoliverau | fungi: ^^ | 23:43 |
| jeblair | jhesketh: i was more inclined to say "let's just let the javascript experts put whatever they want there" a while back, but that's how we ended up with two status pages | 23:43 |
| asselin | mattoliverau, fungi out of curiosity, any reason install_puppet.sh is taken from opnstack-infra where the other scripts are scp to the instance? | 23:44 |
| mattoliverau | jeblair: don't worry, now jhesketh is the javascript expert :P | 23:44 |
| fungi | mattoliverau: are you our new openstackgerrit bot? ;) (i'll go find out where it wandered off to) | 23:44 |
| jhesketh | mattoliverau: oh goodness no! | 23:45 |
| mgagne | clarkb: As you might already know from previous conversations, we are using Jenkins to tag and perform most tasks which would normally be done by a human release manager. It partially explains why our tags aren't signed yet. | 23:45 |
| mattoliverau | fungi: I could be.... but I don't think I'm as efficient :P | 23:45 |
| jeblair | mattoliverau: until he realizes he doesn't actually like javascript. then the status page is maintanerless again... | 23:45 |
| fungi | mattoliverau: that script is also used in other contexts, mainly | 23:45 |
| *** thuc has joined #openstack-infra | 23:45 | |
| jhesketh | jeblair: this was meant to be an afternoon of adding zuul results to gerrit.. .it's now taken weeks | 23:45 |
| jhesketh | I already know I don't like javascript | 23:45 |
| clarkb | jhesketh: :/ | 23:45 |
| clarkb | jhesketh: sorry, I did like the results of what I saw last I fetched it locally | 23:46 |
| clarkb | fwiw | 23:46 |
| mattoliverau | jhesketh: well thanks for taking one for the team :) | 23:46 |
| jhesketh | well it's not terrible, I just miss python | 23:46 |
| jhesketh | tanks clarkb :-) | 23:46 |
| jhesketh | *thanks | 23:46 |
| fungi | jhesketh: i'm sure it misses you too | 23:46 |
| *** dstanek has quit IRC | 23:46 | |
| jhesketh | fungi: it's a sad time for our relationship, but we'll make it | 23:46 |
| jeblair | jhesketh: yeah, so i worry that if we get to the point where the people hacking on zuul like working on the ui even less than we do now, we'll eventually end up in a worse place than we are now | 23:47 |
| jhesketh | yeah those are fair concerns | 23:47 |
| fungi | mattoliverau: once that merges, i'll try spinning up another image to double-check | 23:47 |
| mattoliverau | fungi: me too in my test environment :) | 23:47 |
| jhesketh | jeblair: so I can rearrange my changes to before all of this refactoring, but I don't really want to do so if we'll eventually refactor | 23:48 |
| jhesketh | who's likely to know enough about the issues etc? | 23:48 |
| jhesketh | or somebody that might care | 23:48 |
| jeblair | jhesketh: i hate to impede your work, but all things being equal, i'd probably ask mordred to make a case for us all learning 5 new tools in a language we hate and reading a 1300 line patch | 23:48 |
| jeblair | jhesketh: we probably should get mordred and krotscheck to help us evaluate it | 23:49 |
| jhesketh | jeblair: no worries, I'd rather slow down than do the wrong thing for the project/long term | 23:49 |
| jeblair | krotscheck is the expert, mordred is the proponent, and we know the roadmap | 23:49 |
| *** mspreitz has quit IRC | 23:49 | |
| krotscheck | I’m not seeing a patch in the scrollback, is there something I should be looking at? | 23:50 |
| jeblair | krotscheck: we're talking about https://review.openstack.org/#/c/82307/4 | 23:50 |
| fungi | is this a potential inflection point for spinning the zuul status ui into a separate project and growing more api separation between it and zuul? | 23:50 |
| krotscheck | Oh, that one. | 23:50 |
| *** hashar has quit IRC | 23:50 | |
| jeblair | fungi: i think zuul's api needs to be something other than read-only before that's truly useful | 23:51 |
| krotscheck | jeblair: A language you hate? ;) | 23:51 |
| jeblair | krotscheck: hate may be strong. i'll own to dislike though. :) | 23:51 |
| fungi | jeblair: granted | 23:51 |
| *** e0ne has joined #openstack-infra | 23:52 | |
| jeblair | krotscheck, jhesketh: anyway, i don't want to burn cycles on this now, i think we need mordred around to figure this out | 23:53 |
| jhesketh | fungi: that is something I'd like to see long term.. I don't like zuul having a wsgi app | 23:53 |
| krotscheck | Wow, that patch got a lot bigger. | 23:53 |
| jhesketh | yep +1 | 23:53 |
| jhesketh | krotscheck: then I've gone and moved it around and fixed it : https://review.openstack.org/#/c/84069/ | 23:53 |
| fungi | jeblair: just basically wondering whether that's where the ui work should first focus prior to something like mordred's proposed patch, and then once the api is robust enough to support such separation that separate suite of tools is less onerous to the backend server project | 23:54 |
| jeblair | fungi: everything about what you just said makes me thing more and more that the right answer is to ignore everything until we just fix the thing we set out to fix. :) | 23:55 |
| *** e0ne has quit IRC | 23:55 | |
| *** mrodden has joined #openstack-infra | 23:56 | |
| fungi | jeblair: yeah, i basically feel the same | 23:56 |
| *** flaper87 is now known as flaper87|afk | 23:57 | |
| krotscheck | fungi, jeblair: And maybe by then the node people will have their shit in order and have a recent set of packages available. | 23:58 |
| jeblair | krotscheck: anything's possible! :) | 23:59 |
| jeblair | i have to run for a bit | 23:59 |
| *** aarongr_ has quit IRC | 23:59 | |
| *** mspreitz has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!