| clarkb | Thelo: grep for LIBS_FROM_GIT | 00:00 |
|---|---|---|
| clarkb | you will see its overridden | 00:00 |
| *** zz_dimtruck is now known as dimtruck | 00:00 | |
| jeblair | | local_gb | 915 | | 00:01 |
| jeblair | i get that from nova-hypervisor show | 00:01 |
| jeblair | er nova hypervisor-show | 00:01 |
| jeblair | also: | 00:01 |
| jeblair | | disk_available_least | 835 | | 00:01 |
| jeblair | | free_disk_gb | 865 | | 00:01 |
| jeblair | | local_gb_used | 50 | | 00:01 |
| asselin_ | Thelo, yes, what clarkb said, it's get soverride at 2016-01-12 01:28:48.742 | +++ LIBS_FROM_GIT= | 00:02 |
| jeblair | 915-50=865; i don't know what disk_available_least is | 00:02 |
| Thelo | asselin: mmm maybe my understanding of LIBS FROM GIT is wrong the, in my case I dont want the upstram but a specific pachtset, is it doable? | 00:03 |
| jeblair | ah, it's available space including an overcommit allowance. | 00:03 |
| jeblair | at any rate, it's not looking like different classes of storage are bexing exposed through the parts of the nova api i'm seeing | 00:04 |
| clarkb | greghaynes: https://jenkins03.openstack.org/job/gate-nodepool-python27/126/console otherwise I Think it is looking good | 00:04 |
| asselin_ | Thelo, yes, but the issue is that you setting is getting ignored. | 00:04 |
| *** bharathm has joined #openstack-infra | 00:04 | |
| *** rbrndt has quit IRC | 00:05 | |
| clarkb | LIBS_FROM_GIT says install the lib from the git repo instead of pip. The problem here is that that variable is empty | 00:05 |
| *** baoli has quit IRC | 00:07 | |
| Thelo | clarkb: true, so at which point should I specify LIBS FOR GIT so it is not ignored? | 00:07 |
| greghaynes | clarkb: fun | 00:08 |
| clarkb | Thelo: well if you just set it once you'd be fine but it looks like its being set twice the second time its empty | 00:08 |
| *** dims has joined #openstack-infra | 00:09 | |
| *** rkukura has quit IRC | 00:10 | |
| asselin_ | Thelo, export DEVSTACK_PROJECT_FROM_GIT= | 00:10 |
| *** tzn has quit IRC | 00:10 | |
| Thelo | asselin: Where is that line ? something to add? | 00:11 |
| asselin_ | Thelo, you can include that in your jenkins job | 00:12 |
| Thelo | asselin: ok ; I will try | 00:12 |
| asselin_ | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n301 | 00:12 |
| *** bcourt has joined #openstack-infra | 00:12 | |
| asselin_ | Thelo, this is the line ^^ | 00:12 |
| Thelo | asselin: what does it do? | 00:13 |
| Thelo | asselin: O i get it | 00:13 |
| asselin_ | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n204 | 00:13 |
| Thelo | asselin, clarkb: thanks | 00:13 |
| asselin_ | this is where it's actually set | 00:13 |
| *** ddieterly has quit IRC | 00:13 | |
| *** salv-orl_ has quit IRC | 00:16 | |
| *** piet has quit IRC | 00:17 | |
| *** salv-orlando has joined #openstack-infra | 00:17 | |
| *** piet has joined #openstack-infra | 00:20 | |
| jeblair | jhesketh: no, the zuul behavior you describe doesn't ring a bell | 00:20 |
| *** bpokorny_ has quit IRC | 00:20 | |
| jeblair | jhesketh: and sorry, i've gotten swamped this afternoon and haven't had a chance to discuss ansible yet | 00:20 |
| *** bpokorny has joined #openstack-infra | 00:21 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:21 | |
| jeblair | jhesketh: but really quickly, the gist is that i think it will be safer if we invoke ansible as a subprocess rather than using it directly in python | 00:21 |
| *** eliqiao has quit IRC | 00:22 | |
| jeblair | jhesketh: the python api is not very well documented, and seems mostly to exist for the purpose of driving ansible's command line clients, and they make some significant assumptions in doing that. | 00:23 |
| *** eliqiao has joined #openstack-infra | 00:23 | |
| *** sdake has joined #openstack-infra | 00:24 | |
| jeblair | jhesketh: so i'm going to revise my latest v3 change to do that and then turn it over to you | 00:24 |
| *** jsavak has quit IRC | 00:24 | |
| jhesketh | jeblair: all good.. does that sound like incorrect behavour (re not rebasing onto the newer patchset) | 00:25 |
| jhesketh | jeblair: good to know re the ansible lib. Once the zuul worker sets up the env and everything it should be a pretty simple call out to ansible | 00:26 |
| jhesketh | ie ansible will do all the heavy lifting of the job itself so I'm hoping that part is straight forward | 00:26 |
| *** ZZelle_ has quit IRC | 00:28 | |
| *** rkukura has joined #openstack-infra | 00:29 | |
| *** tongli has joined #openstack-infra | 00:30 | |
| *** Swami has joined #openstack-infra | 00:32 | |
| *** bharathm has quit IRC | 00:33 | |
| *** bharathm has joined #openstack-infra | 00:34 | |
| *** otter768 has joined #openstack-infra | 00:35 | |
| *** tongli has quit IRC | 00:40 | |
| *** otter768 has quit IRC | 00:40 | |
| *** tongli has joined #openstack-infra | 00:40 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add more details about Emotional Jenkins plugin to docs https://review.openstack.org/265941 | 00:43 |
| *** dimtruck is now known as zz_dimtruck | 00:44 | |
| mtreinish | fungi: is it still running? This is pretty slow :) | 00:45 |
| fungi | mtreinish: still running | 00:46 |
| dims | oh man looking at zuul i don't even want to ship oslo releases this week | 00:48 |
| clarkb | its actually not that bad | 00:49 |
| *** rkukura has quit IRC | 00:49 | |
| clarkb | dims: what is the concern? | 00:49 |
| dims | clarkb : gate depth, just in case i break something else | 00:50 |
| tonyb | So I'm trying to help a friend. He has a change (in os-brick) that depnds-on changes to requirements and privsep. Looking at the logs from the job zuul correctly set the ZULL_CHANGES envvar. The requirements change was pulled in such that /opt/stack/new/requirements was the right git SHA but oslo.provsep was just installed from pip. my gut feel is that I need to ass oslo.privsep to $PROJECTS for os-brick but I can't verify that. Does it sound correct | 00:50 |
| *** rkukura has joined #openstack-infra | 00:50 | |
| tonyb | https://review.openstack.org/#/c/258252 is the change in question. | 00:51 |
| dims | tonyb : use LIBS_FROM_GIT=oslo.privsep | 00:51 |
| jeblair | jhesketh: it sounds like an error -- uploading a new patchset should eject that item from the queue | 00:51 |
| jhesketh | jeblair: it was a new patchset to a change above the one that wouldn't enter the gate | 00:51 |
| tonyb | dims: but I only want it iff there is a Depends-On | 00:51 |
| jeblair | jhesketh: i may not understand the situation :) | 00:52 |
| clarkb | tonyb: adding it to PROJECTS is a necessary step, but it is not the only required step, you need libs from git too | 00:52 |
| jhesketh | yeah difficult to explain sorry | 00:52 |
| jhesketh | jeblair: https://review.openstack.org/#/c/260628/3 | 00:52 |
| jhesketh | jeblair: 260628 depends on 260625/4 but it was 260625/6 that merged | 00:53 |
| jhesketh | 260628/3 in the related changes section links to gitweb rather than to the change (which is a new thing to me) | 00:53 |
| *** markmcclain has quit IRC | 00:53 | |
| jhesketh | in order to get it into the gate we had to rebase 260628 onto 260625/6 (well in reality since 260625/6 had merged I rebased onto master) | 00:54 |
| dims | tonyb : looking | 00:54 |
| jhesketh | jeblair: you can kinda see what was going on in the comment section | 00:54 |
| *** bcourt has quit IRC | 00:55 | |
| tonyb | dims: Thanks. I'm struggling to find the linkage between what zuul decides is required, based on Depends-On and how the node is setup. | 00:55 |
| *** zz_dimtruck is now known as dimtruck | 00:55 | |
| *** Sukhdev has quit IRC | 00:55 | |
| clarkb | tonyb: all zuul is going to do is make sure the code is merged proprely and provide a zuul ref with that change | 00:55 |
| *** amuller has quit IRC | 00:55 | |
| clarkb | tonyb: your job(s) then need to properly checkout that zuul ref and install from source | 00:55 |
| *** markmcclain has joined #openstack-infra | 00:55 | |
| clarkb | PROJECTS list ensures that the listed projects are checked out properly and LIBS_FROM_GIT installs that lib from source | 00:56 |
| jeblair | jhesketh: looking | 00:56 |
| jhesketh | jeblair: let me know if you want me to retry that bad explaination | 00:56 |
| *** tlian has quit IRC | 00:57 | |
| * jeblair opens that patch in gertty to get correct (utc) times | 00:57 | |
| tonyb | clarkb: okay but I don't *always* want LIBS_FROM_GIT, I only want it if this change Depends-On one from oslo.privsep | 00:57 |
| tonyb | clarkb: I'm not finding eveidnece that other projects have had to do that checking so I thought I was missing something | 00:58 |
| clarkb | tonyb: then write a one off change to do it | 00:58 |
| clarkb | thats probably the simplest way to do it | 00:58 |
| clarkb | (but this is the problem with not doing things the old way) | 00:59 |
| clarkb | that said, you probably don't want both changes ot need atomic merging | 00:59 |
| tonyb | clarkb: woudl the change be to devstack-gate? what is the old way? | 00:59 |
| clarkb | tonyb: the old way was we installed everything fomr source | 00:59 |
| clarkb | tonyb: I would probably make the change to devstack but either one works | 00:59 |
| dims | tonyb : the way i would do this with current setups...is as follows... 1) review for oslo.privsep if needed 2) review for os-brick if needed 3) review for devstack with depends-on for reviews 1 and 2 and LIBS_FROM_GIT set to both oslo.privsep and os-brick | 01:01 |
| dims | then run the tests in review #3 and keep adjusting #1 and #2 till you are happy | 01:01 |
| *** eil397 has quit IRC | 01:02 | |
| *** ybathia has quit IRC | 01:03 | |
| clarkb | tonyb: I think the idea behind having this setup is it forces you to not make changes that need multiple things to merge all toegher or release togethre | 01:03 |
| dims | tonyb : the other trick i use is modify my tox.ini to run "pip install -q -U -e git+https://github.com/openstack/oslo.cache.git#egg=oslo.cache" | 01:04 |
| clarkb | tonyb: so privsep should support the old and new thing, then you update brick, then you can deprecate the ol thing | 01:04 |
| *** tlian has joined #openstack-infra | 01:04 | |
| clarkb | but either way what dims describes will work to test them prior to any releases | 01:04 |
| tonyb | clarkb: I mostly get that but ... | 01:05 |
| jeblair | (really there should be some kind of job on the oslo lib that exercises this) | 01:05 |
| tonyb | clarkb: we're trying to develop the 'new thing' in privsep and verify that brick works, and also trying to avoid lots of broken releases of privsep | 01:05 |
| clarkb | ya if this is an ongoing thing a job that just does it may be worthwhile | 01:06 |
| dims | clarkb : so it's master of oslo.privsep + master of os-brick tested with cinder/nova | 01:06 |
| clarkb | dims: no thats not quite right | 01:06 |
| clarkb | dims: its the proposed change of privsep + the proposed change of brick against cinder/nova | 01:07 |
| *** piet has quit IRC | 01:07 | |
| dims | clarkb : right :) | 01:07 |
| dims | master + review instead of library from pypi | 01:07 |
| clarkb | well it doesn't have to be master | 01:07 |
| dims | clarkb : explaining tonyb's situation. but yes you are right | 01:08 |
| tonyb | dims: ate this point there is no cinder/nova in the picture it's just os-brick using privsep | 01:08 |
| clarkb | so ya I think your two options are make a temporary job that just does this for your changes as you go or push a change to devstack to configure it that way | 01:08 |
| jeblair | ++temp job | 01:08 |
| dims | tonyb : then you can use the 2nd trick. | 01:08 |
| dims | tonyb : "pip install -q -U -e git+https://github.com/openstack/oslo.cache.git#egg=oslo.cache" in tox.ini just before you run tests | 01:09 |
| jeblair | and some day, someone will write a change to zuul to launch jobs on request | 01:09 |
| tonyb | dims: sure. but how do I do that only when there is a Depnds-On | 01:09 |
| jeblair | tonyb: if a job changed behavior based on a depends on, then you would be able to break a gate | 01:10 |
| *** tongli has quit IRC | 01:10 | |
| *** ekarlso has quit IRC | 01:10 | |
| jeblair | tonyb: the next change after that might not have a depends on and would be testing different code | 01:10 |
| jeblair | so that's not a pattern we should use | 01:10 |
| *** tzn has joined #openstack-infra | 01:10 | |
| *** tongli has joined #openstack-infra | 01:11 | |
| *** tongli has quit IRC | 01:11 | |
| tonyb | wait what? | 01:11 |
| clarkb | the issue here is what is desired is an integration test between two repos which does not exist today | 01:11 |
| *** annegent_ has joined #openstack-infra | 01:11 | |
| tonyb | we use depends-on often | 01:11 |
| *** tongli has joined #openstack-infra | 01:11 | |
| tonyb | how is this different? | 01:11 |
| clarkb | the best way to solve that problem is to make the integration test | 01:11 |
| jeblair | clarkb: ++ | 01:11 |
| clarkb | tonyb: depends on requires the individual jobs to checkout and install the correct code | 01:13 |
| *** annegentle has quit IRC | 01:13 | |
| dims | clarkb : when you do a Depends-On does the current setup pull down code from git if it is not present in PROJECTS? | 01:13 |
| clarkb | dims: no | 01:13 |
| dims | i am guessing not | 01:13 |
| tonyb | I don't mean to be dense. but I'm celarly missing somethign important | 01:13 |
| dims | tonyb : see ^^^ | 01:13 |
| clarkb | tonyb: all depends on does in zuul is make a zuul ref | 01:13 |
| *** tlian has quit IRC | 01:13 | |
| clarkb | tonyb: something needs to grab the zuul ref and install it. Devstack-gate and zuul-cloner will both do it for you | 01:13 |
| *** tlian has joined #openstack-infra | 01:13 | |
| tonyb | clarkb: okay so is the zuul ref a cross project thing? | 01:14 |
| clarkb | tonyb: yes each project in that dependency gets the same ref in their repo | 01:14 |
| tonyb | clarkb: okay. | 01:14 |
| *** gongysh has joined #openstack-infra | 01:14 | |
| tonyb | clarkb: so I *think* just adding oslo.privsep to PROJECTS will work | 01:15 |
| clarkb | so in this case you need a job that checks out the zuul ref for privsep and brick, then you need to install them from that source, then you run whatever tests against them | 01:15 |
| jeblair | tonyb: a zuul ref is a potential future state of the world for all involved repositories -- including ones specified by depends on. | 01:15 |
| dims | tonyb : the other thing i do is work on my github repo and use the pip install -U trick to test | 01:15 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder https://review.openstack.org/263349 | 01:15 |
| tonyb | then devstack-gate will use zuul-cloner to grab the zuul_ref | 01:15 |
| tonyb | that's the part that's missing. | 01:15 |
| clarkb | tonyb: it will do the equivalent thing yes, but what it probably won't do is install privsep from that location | 01:15 |
| clarkb | most(all) libs are installed from pip | 01:16 |
| * tonyb goes to check that bit | 01:16 | |
| gus | ok, so (being dense again) what does Depends-On actually help me with, if not this exact situation? | 01:16 |
| clarkb | gus: it does help this exact situation | 01:16 |
| *** tzn has quit IRC | 01:16 | |
| clarkb | gus: but you still need an integration test between the two repos | 01:16 |
| gus | .. except I need to predeclare all the combinations of Depends-On that I might use? | 01:16 |
| clarkb | something that says if I have this repo and that repo grab them both and test them together | 01:16 |
| *** mriedem has joined #openstack-infra | 01:16 | |
| clarkb | no, so thats my point earlier about how and why oslo no longer tests everything from source all together anymore | 01:17 |
| clarkb | there was a conscious desecision that these libs should have interfaces that are more stable than that | 01:17 |
| clarkb | there may be exceptions, but in those cases you will need a different job | 01:18 |
| *** doug-fish has quit IRC | 01:18 | |
| gus | clarkb: I'm all for that, and I'd be happy if the downstream patches couldn't merge until all the depends-on were worked out - but this makes development really hard :/ | 01:18 |
| *** rkukura has quit IRC | 01:18 | |
| jeblair | the oslo lib developers have _intentionally_ decided that new features should be developed in oslo first, then released, then they can begin to be used in other projects. | 01:18 |
| jeblair | and if those features don't work right, they will need to be adjusted in oslo | 01:18 |
| jeblair | and then oslo released again | 01:18 |
| gus | I basically can't iterate on an os-brick CI bot thinks of my change(s) without doing pypi release for every one. | 01:18 |
| clarkb | gus: you can... | 01:19 |
| clarkb | you just need a new job | 01:19 |
| gus | sure :P | 01:19 |
| clarkb | because the existing job setup has intentionally set it up such that brick changes will test against release privsep and privsep changes will test against release brick | 01:19 |
| *** doug-fish has joined #openstack-infra | 01:19 | |
| dims | gus: you can review notes above from me on how i do this currently | 01:19 |
| clarkb | (its worth noting that infra iddn't enforce this, oslo did it) | 01:20 |
| clarkb | (we just run the jobs that oslo decided they wanted/needed to run) | 01:20 |
| jeblair | right, this is not a CI limitation, it's a choice the project developers have made | 01:20 |
| gus | oh yeah, no blame intended - I'm just a bit confused how people work in this environment. | 01:20 |
| clarkb | if oslo wants to install everything fomr source again that totally doable | 01:20 |
| clarkb | and its how we tested everything for about 4 years | 01:20 |
| dims | gus: i've shared all of my tips :) | 01:20 |
| dims | clarkb : jeblair : sdague will have plenty to say on that :) | 01:21 |
| jeblair | dims: i wish sdague were here to explain this right now | 01:21 |
| tonyb | okay so would a devstack-gate change that iterated over ZUUL_CHNAGES and added things to $PROJECTS and LIBS_FROM_GIT be laughed at? | 01:21 |
| jeblair | we have a system that excels at testing coordinated releases of software, but we only use it for a subset of our projects. | 01:21 |
| jeblair | tonyb: it is important that a job have a consistent behavior -- it needs to either install each of its dependencies from source or pip every time it runs | 01:22 |
| jeblair | tonyb: we don't care which it does, only that it does the same thing for each dependency each time | 01:22 |
| jeblair | tonyb: it can be all source, all pip, or a mixture | 01:22 |
| jeblair | tonyb: but it can't change from one run to the next | 01:23 |
| gus | dims: so my challenge with os-brick is that I don't have any of the hardware to actually test most of these codepaths. | 01:23 |
| jeblair | tonyb: (of course, you can change the job configuration, but that's a static change) | 01:23 |
| jeblair | gus: are you working on the same thing as tonyb? | 01:23 |
| tonyb | jeblair: run to run of the same review or ro run to run of the same repo? | 01:23 |
| gus | So I'm looking closely at the 3rd party CI bots. It seems there's no way to actually get them to run my unreleased privsep code. | 01:23 |
| *** eil397 has joined #openstack-infra | 01:23 | |
| asselin_ | gus, there should be some 3rd party cis running to get more coverage in os-brick | 01:23 |
| dims | gus : remember that tox will install stuff from requirements file too which is a pypi version | 01:24 |
| jeblair | tonyb: run to run of the same _job_. | 01:24 |
| tonyb | jeblair: yeah gus is the friend I'm tryign to help | 01:24 |
| jeblair | hi gus :) | 01:24 |
| tonyb | jeblair: okay | 01:24 |
| gus | jeblair: yes (or rather, he's bravely investigating my problem for me) | 01:24 |
| jeblair | gus, tonyb: clarkb's point earlier is the key -- you have found a cross-project integration test that is missing | 01:24 |
| *** eliqiao has quit IRC | 01:24 | |
| asselin_ | I think Thelo was asking about that too, and he discusseed the issue with hemna over in the cinder channel | 01:25 |
| dims | jeblair : os-brick does not depend on oslo.privsep today (if you see master) | 01:25 |
| asselin_ | gus, tonyb have you talked to hemna yet? | 01:25 |
| tonyb | so I have to create a new job in in brick (and later nova/cinder) that pulls brick and privsep from source (as defined by the ZUUL_REF) | 01:25 |
| jeblair | gus, tonyb: for this situation, it seems like it would be worthwhile to add a job to perform that test, at least as long as this development effort runs, and then it can be removed when no longer relevant. | 01:25 |
| gus | jeblair: ok, but surely this is true for just about every oslo library? and yet there was a decision not to just list all oslo libraries in this situation. | 01:25 |
| dims | tonyb : yes new experimental job for os-brick that adds oslo.privsep in PROJECTS. | 01:26 |
| tonyb | asselin_: gus has, I just *thought* I understood this and could help him | 01:26 |
| *** bharathm has quit IRC | 01:26 | |
| tonyb | dims: okay experimental for now later non-voting I guess | 01:26 |
| dims | tonyb : when you do that if you have Depends-On then the correct review is pulled down, you will still need to install it with pip commands in tox.ini | 01:27 |
| dims | against the downloaded directory | 01:27 |
| gus | asselin_: I last did a few days ago.. | 01:27 |
| asselin_ | tonyb, gus please talk to hemna in #openstack-cinder. He's an os-brick core and familiar with this challenge and can help you find a path forward | 01:27 |
| openstackgerrit | lifeless proposed openstack-infra/infra-manual: Update the manual to reference -constraints jobs. https://review.openstack.org/255656 | 01:27 |
| *** Daisy has joined #openstack-infra | 01:28 | |
| *** mtanino has quit IRC | 01:28 | |
| *** ccrouch has quit IRC | 01:29 | |
| jeblair | jhesketh: 2016-01-12 04:57:59,610 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack/horizon master 260628,3> | 01:29 |
| tonyb | okay I've never used the experimental queue before. I know how to trigger it but where do I get the results/logs? | 01:29 |
| jeblair | jhesketh: zuul saw the jenkins+1 but did not report self.log.info("Adding %s, %s to %s" % (project, change, pipeline)) | 01:29 |
| jeblair | jhesketh: which suggests that it did not match the pipeline trigger for gate | 01:30 |
| jeblair | jhesketh: aha | 01:30 |
| jhesketh | jeblair: right, it also didn't enter the gate when a +W was just added either | 01:30 |
| *** tlian has quit IRC | 01:30 | |
| jeblair | jhesketh: it's because jenkins already had a +1 | 01:30 |
| fungi | tonyb: they get posted to the change where you leave a check experimental comment | 01:30 |
| jeblair | jhesketh: yeah, so i've been looking at the wrong event :) | 01:30 |
| lifeless | 3rd-party should honour depends-on | 01:30 |
| lifeless | if it doesn't, its a bug, no? | 01:30 |
| tonyb | fungi: ok. | 01:31 |
| jeblair | lifeless: we've not required that they run zuul | 01:31 |
| * tonyb goes to look for that | 01:31 | |
| jeblair | lifeless: many do, but some don't | 01:31 |
| fungi | lifeless: also some of them are only set up to test changes to specific repos and need to be able to skip dependencies on changes to projects they don't have defined in zuul | 01:31 |
| lifeless | jeblair: Oh, I wasn't assuming running-zuul, just that they test the same combinations upstream does | 01:32 |
| jeblair | tonyb, gus: yeah if you are talking about third-party ci systems, all bets are off. | 01:32 |
| lifeless | jeblair: but I can see how that is coupled | 01:32 |
| tonyb | jeblair: no just the infra run CI | 01:32 |
| jeblair | lifeless: afaik, depends-on is only implemented by zuul. yes. :) | 01:32 |
| *** Keedya_ has joined #openstack-infra | 01:33 | |
| lifeless | fungi: that doesn't make sense; if they are testing changes to cinder, but oslo.logging is a depends-on of a cinder change, they need the depends-on to be able to report usefully | 01:33 |
| *** doug-fish has quit IRC | 01:33 | |
| lifeless | fungi: perhaps I misinterpreted what you said | 01:33 |
| jeblair | lifeless: agreed in theory; i think in practice generally the 3rd-party ci test scope may be considered less thorough than what we would consider ideal | 01:34 |
| gus | jeblair: hrm, yes of course. | 01:35 |
| gus | (perhaps 3rd party bots that "don't support" Depends-On, should support it enough to give up on any change that includes a Depends-On?) | 01:35 |
| *** Keedya_ has quit IRC | 01:36 | |
| fungi | lifeless: they often don't define the complete set of all the hundreds of repos that we do in our zuul config, and zuul can't create refs for repos it doesn't know about | 01:36 |
| fungi | so a depends-on to a change in an undefined repo gets handled as a no-op at this point (until a few months ago it caused zuul to throw an exception and not run any jobs for the depending change instead) | 01:37 |
| jeblair | (incidentally, something i want to change in v3) | 01:37 |
| *** dims has quit IRC | 01:38 | |
| lifeless | fungi: jeblair: thanks for the details. | 01:38 |
| lifeless | gus: I suspect contacting the third party ci in question perhaps ? | 01:39 |
| *** hdd has quit IRC | 01:39 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump versions for each keystone library https://review.openstack.org/266629 | 01:39 |
| jeblair | 2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7f06dd0bc6d0 260628,3>: | 01:40 |
| jeblair | 2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f06dd0bc6d0 260628,3> needs change <Change 0x7f0827250890 260625,4>: | 01:40 |
| jeblair | 2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Needed change is not the current patchset | 01:40 |
| jeblair | 2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Failed to enqueue changes ahead of <Change 0x7f06dd0bc6d0 260628,3> | 01:40 |
| jeblair | jhesketh: ^ | 01:40 |
| tonyb | lifeless: we're not targetting 3rd party ci just the infra's CI | 01:40 |
| lifeless | tonyb: to get test results on os-brick for oslo.privsep? I may have misunderstood the conversation | 01:41 |
| jeblair | jhesketh: i think that is in accord with what you described. | 01:41 |
| gus | lifeless: I expect so. I'm conscious of the fact that privsep is just an unimportant one amongst many project dependencies. So it seems unreasonable to ask for things that are privsep+$project-specific just to save me some time on development iteration. | 01:41 |
| *** eliqiao has joined #openstack-infra | 01:42 | |
| jeblair | gus: the infrastructure is here to help everyone save time (and mistakes) in developer iteration :) | 01:42 |
| gus | jeblair: you'll excuse me for not feeling like that at the moment ;) | 01:42 |
| lifeless | gus: its a rather more invasive change than many | 01:42 |
| jeblair | gus: don't you have widespread support from those in channel for adding the job that will help you do your work? | 01:43 |
| gus | jeblair: I presume so. I'll create the change and see what happens ;) | 01:44 |
| gus | (My alternative is to work on something else while I wait for the review+merge+release train) | 01:45 |
| fungi | as to the reason we ended up with most jobs testing only with their released library dependencies, that was to mitigate a previous breakage risk where people were releasing unusable things which expected unreleased behavior in their dependencies | 01:45 |
| fungi | and didn't notice because the jobs were passing | 01:46 |
| gus | fungi: yeah, I can appreciate that risk. | 01:47 |
| *** piet has joined #openstack-infra | 01:47 | |
| clarkb | right it forced us away from the expectation of atomic releases | 01:47 |
| clarkb | and makes you do more of a rolling system that allows users to upgrade | 01:47 |
| gus | If we only had infinite development resources, then we could come up with a separate tests for "in development" vs "ready for release" or somesuch. | 01:48 |
| gus | :P | 01:48 |
| jeblair | gus: some repos have both | 01:48 |
| fungi | yep, which is a reasonable compromise even if still incomplete (as opposed to trying to test every possible combination of latest release and branch tip of all dependencies) | 01:49 |
| fungi | ultimately people are going to either be installing from latest source or latest releases anyway | 01:49 |
| *** amrith is now known as _amrith_ | 01:50 | |
| fungi | and not mixing and matching all over the place | 01:50 |
| gus | fungi: right, it's only the out-of-process dependencies that need to coexist during migration (assuming you can't stop-the-world). | 01:51 |
| fungi | so 1. make sure it works with things that have been released, and 2. make sure it works with things that have been merged but not yet released (and the interaction between proposed changes of two dependencies can be tested by the latter with change dependencies declared) | 01:51 |
| clarkb | the mix and matching comes from mixing control planes | 01:51 |
| clarkb | it is very common to collocate many services | 01:52 |
| clarkb | but not to upgrade them all at the same time | 01:52 |
| gus | clarkb: meh, "venvs" or some other jedi hand waving. | 01:52 |
| fungi | gus: easier now than it used to be, but we still have plenty of packages which import from each other, so you can't silo them all | 01:53 |
| gus | docker containers, chroots, etc. | 01:53 |
| *** chlong is now known as chlong-afk | 01:53 | |
| *** gongysh has quit IRC | 01:54 | |
| clarkb | I dunno I think its a good thing to be disciplined about | 01:54 |
| clarkb | sure you can work around it but the idea that you have to is silly | 01:54 |
| fungi | panaceas, magic bullets | 01:54 |
| fungi | rainbow vomiting unicorns | 01:54 |
| fungi | all equally valid solutions to the problem | 01:55 |
| Clint | unicorn pancreas | 01:55 |
| clarkb | especailly if we eve rwan to address the "openstack is hard to deploy problem" | 01:56 |
| *** rkukura has joined #openstack-infra | 01:57 | |
| fungi | Clint: with some fava beans and a nice chianti | 01:57 |
| Clint | too bad i'm full | 01:58 |
| jeblair | jhesketh: i just re-checked that gerrit will (thankfully) not merge a change to a branch whose parent is not in the branch | 01:58 |
| jeblair | jhesketh: https://review-dev.openstack.org/#/c/5403/ | 01:58 |
| jeblair | jhesketh: so zuul is doing the correct thing by preventing the change from being enqueued into the gate pipeline since it has no possibility of actually merging at the end | 01:59 |
| *** eil397 has quit IRC | 01:59 | |
| asselin_ | hey any infra-root available to create a feature branch for me? https://review.openstack.org/#/admin/projects/openstack-infra/ciwatch,branches | 01:59 |
| jeblair | jhesketh: in short, i believe zuul's behavior is correct | 01:59 |
| fungi | jeblair: is that new in 2.11? it sure did in 2.8 | 01:59 |
| jeblair | fungi: well, at least a change whose parent is outdated | 02:00 |
| jeblair | fungi: well, at least a change whose parent _change_ is outdated | 02:00 |
| jeblair | fungi: i haven't tested the 'change whose parent isn't even in gerrit' case :) | 02:00 |
| fungi | jeblair: ahh, okay. the other issue was that it would fast-forward if the parent was on another branch and had no merge conflicts | 02:00 |
| jeblair | Change cannot be merged due to unsatisfiable dependencies. | 02:00 |
| jeblair | The following dependency errors were found: | 02:00 |
| jeblair | Depends on patch set 1 of I7af6e09b, however the current patch set is 2. | 02:00 |
| jeblair | fungi: that ^ | 02:00 |
| fungi | yep. that matches my expectations | 02:01 |
| *** bardia has quit IRC | 02:01 | |
| jeblair | fungi: yep. i believe that was the behavior in 2.8 as well, and that's why zuul has enforced that for some time. | 02:01 |
| jeblair | fungi: i'm just being thorough with jhesketh's report :) | 02:01 |
| *** piet has quit IRC | 02:01 | |
| jeblair | i did not want to make any assumptions about behavior that may have changed in gerrit | 02:01 |
| *** rkukura has quit IRC | 02:01 | |
| fungi | we likely still have the issue where if someone proposes a commit from say master to stable/liberty and stable/liberty hasn't diverged so all its commits are in master's history, then gerrit will drag in the intermediate commits | 02:01 |
| fungi | and won't provide much indication in the review ui that it's going to do so | 02:02 |
| fungi | i don't think the bug report about that ever went anywhere | 02:03 |
| *** thorst_ has quit IRC | 02:03 | |
| *** thorst has joined #openstack-infra | 02:04 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: Save the adminPass if returned on server create https://review.openstack.org/266532 | 02:05 |
| fungi | stated differently, gerrit is happy to consider changes mergeable even if their parents aren't in the current branch history, as long as they're in gerrit on another branch and all other mergability requirements are met | 02:05 |
| *** bpokorny_ has joined #openstack-infra | 02:06 | |
| jhesketh | jeblair: shouldn't the gate pull down the newest patchset in that change? | 02:07 |
| jhesketh | jeblair: or are what we're saying is that if a change ahead of itself is updated the child changes need rebasing | 02:07 |
| *** rguillebert has quit IRC | 02:07 | |
| *** bpokorny has quit IRC | 02:09 | |
| *** tongli has quit IRC | 02:09 | |
| Thelo | Hi all, on my CI zuul does not find out any new task (2016-01-13 10:08:33,086 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information.) | 02:10 |
| Thelo | is anyone familiar with that? | 02:10 |
| jhesketh | Thelo: those errors are safe to ignore | 02:10 |
| *** tongli has joined #openstack-infra | 02:10 | |
| jhesketh | it's because of new events in a newer gerrit | 02:10 |
| *** rkukura has joined #openstack-infra | 02:10 | |
| jhesketh | there is a patch up to deal with them | 02:10 |
| *** bpokorny_ has quit IRC | 02:10 | |
| jhesketh | not finding a new task would be unrelated | 02:10 |
| *** tzn has joined #openstack-infra | 02:13 | |
| *** Swami has quit IRC | 02:13 | |
| *** thorst has quit IRC | 02:14 | |
| *** tongli has quit IRC | 02:15 | |
| Thelo | jhesketh: is there a delay between the git review command and the moment where the log should appear in zuul/debug.log? | 02:15 |
| clarkb | asselin_: did you get your branch? | 02:15 |
| clarkb | fungi: ^ anything special you want ot instruct them on using a feature branch? | 02:16 |
| *** Qiming has joined #openstack-infra | 02:16 | |
| fungi | clarkb: nah, i referred them to our lovely infra manual bit about merge commits | 02:16 |
| clarkb | kk making branch now | 02:16 |
| fungi | i thought i already did weeks ago | 02:16 |
| clarkb | fungi: not on ciwatch | 02:17 |
| clarkb | asselin_: feature/unit-tests? | 02:17 |
| fungi | oh, ciwatch. i was thinking of the jjb refactoring feature branch | 02:17 |
| clarkb | ah | 02:17 |
| clarkb | I will go ahead and create it and point asselin_ at the manual doc too | 02:17 |
| fungi | thanks | 02:17 |
| * jhesketh just read some missing scrollback | 02:18 | |
| fungi | asselin_: what specifically are you wanting to do with said feature branch? i likely skimmed too lightly | 02:18 |
| clarkb | asselin_: http://docs.openstack.org/infra/manual/drivers.html#feature-branches | 02:18 |
| jhesketh | jeblair, fungi: are you saying gerrit 2.8 would fastfoward onto the newest patchset but 2.11 doesn't? (or am I reading the opposite) | 02:18 |
| fungi | jhesketh: no, 2.8 wouldn't either | 02:18 |
| clarkb | fungi: aiui they are going to refactor ciwatch and add tests and felt that doing such heavy lifting on master would be problematic for keeping master working | 02:18 |
| jhesketh | Thelo: there will be a small delay, but it shouldn't be too long | 02:18 |
| clarkb | fungi: so basically let master keep running as is, then off to the side take a chainsaw to it to refactor and make it testable | 02:19 |
| jhesketh | fungi: hmm okay.. I wonder why I didn't notice this in 2.8.. maybe I just missed it | 02:19 |
| *** tzn has quit IRC | 02:19 | |
| fungi | jhesketh: 2.8 (and likely 2.11 as well) will fast-forward a branch with intermediate commits it knows about | 02:19 |
| jhesketh | fungi: so when a parent moves forward we need to rebase the children? | 02:19 |
| fungi | i wasn't referring to updating patchsets | 02:19 |
| fungi | but yes, just like in 2.8, if you change a parent change to add a new patchset, the child changes become "outdated" in gerrit and need to be rebased onto the new parent | 02:20 |
| *** armax has joined #openstack-infra | 02:21 | |
| jhesketh | okay, for some reason I was under the impression that if a child was behind outdated parents a +W would do the correct thing and put it in the gate | 02:21 |
| jhesketh | good to know, thanks for your help :-) | 02:21 |
| *** mudassirlatif has quit IRC | 02:22 | |
| fungi | yeah, never did i'm pretty positive. i recall dealing with it regularly in 2.8 | 02:23 |
| fungi | and earlier gerrit for that matter | 02:23 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-ciwatch: Add initial implementation to deploy ciwatch https://review.openstack.org/238606 | 02:23 |
| *** zhurong has joined #openstack-infra | 02:24 | |
| EmilienM | hello, can I have a review on https://review.openstack.org/#/c/265447/ please ? it's openstack-infra/os-loganalyze | 02:25 |
| jhesketh | EmilienM: will look | 02:26 |
| EmilienM | jhesketh: thanks! | 02:26 |
| EmilienM | that's not urgent | 02:26 |
| *** piet has joined #openstack-infra | 02:28 | |
| *** zhurong has quit IRC | 02:32 | |
| *** zhurong has joined #openstack-infra | 02:36 | |
| *** otter768 has joined #openstack-infra | 02:37 | |
| *** dimtruck is now known as zz_dimtruck | 02:38 | |
| *** markmcclain has quit IRC | 02:39 | |
| *** markmcclain has joined #openstack-infra | 02:40 | |
| *** otter768 has quit IRC | 02:41 | |
| Thelo | jhesketh: Ok I see the log now, right now I am tryting to trigger basic job each time there is a commit on the sandbox, but although I see the log, no job kick off, do you know what I should look at? | 02:41 |
| jhesketh | Thelo: you have to have a worker (eg jenkins) register with the job name or zuul will return a NOT_REGISTERED error | 02:42 |
| jhesketh | Thelo: and your pipeline triggers need to be set to trigger on a patchset created event | 02:43 |
| Thelo | I do have jenkins how do I know if its register to zuul? | 02:44 |
| jhesketh | Thelo: connecting to gearman using telnet you can use 'status' to see what is registered | 02:45 |
| jhesketh | Thelo: although it sounds like you need to make sure the trigger is firing first | 02:45 |
| Thelo | jhesketh: do you know on which port gearman listens by default? | 02:49 |
| jhesketh | Thelo: 4730 | 02:49 |
| *** piet has quit IRC | 02:50 | |
| *** ccrouch has joined #openstack-infra | 02:50 | |
| Thelo | jhesketh: http://paste.openstack.org/show/483680/ so here if I have the following configuration (https://github.com/thelo-gaultier/itri-disco-ci/blob/master/zuul/layout.yaml) a job should be kick off on jenkins right? | 02:51 |
| *** rossella_s has quit IRC | 02:52 | |
| *** rossella_s has joined #openstack-infra | 02:52 | |
| jhesketh | Thelo: so status shows "total, running, workers" | 02:54 |
| jhesketh | so you have 2 workers registered for the job which is good | 02:54 |
| jhesketh | so the question is why it isn't queuing | 02:54 |
| mmedvede | Thelo: you might need to change last line of layout.yaml to '- dsvm-tempest-my-cinder-driver' | 02:55 |
| Thelo | jhesketh: (thank for the explaination) | 02:55 |
| *** hdd has joined #openstack-infra | 02:56 | |
| Thelo | mmedvede: why? | 02:56 |
| jhesketh | Thelo: so in the logs you should see the event from gerrit and something comparing it to a trigger | 02:56 |
| mmedvede | Thelo: never mind, I did not see the dsvm-tempest-full in your gearman paste | 02:56 |
| jhesketh | Thelo: hmm, you have no source defined in your pipeline... I wonder if that has something to do with it | 02:56 |
| jhesketh | Thelo: did you look at the zuul/etc/zuul.conf.sample ? | 02:57 |
| *** hdd has quit IRC | 02:57 | |
| *** dtardivel has quit IRC | 02:57 | |
| openstackgerrit | Merged openstack/requirements: Bump neutronclient in upper-constraints to 4.0.0 https://review.openstack.org/266042 | 02:58 |
| Thelo | I dont have this file? where should it be? | 02:58 |
| Thelo | jhesketh: in the log I dont see any place where it is "comparing" it to a trigger, do you have a log example? | 03:00 |
| *** zz_dimtruck is now known as dimtruck | 03:01 | |
| mmedvede | Thelo: the conf sample is in zuul repo | 03:01 |
| jhesketh | Thelo: actually you don't need the source, it defaults to gerrit | 03:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 03:02 | |
| *** kushal has quit IRC | 03:02 | |
| jhesketh | Thelo: look for something like "Processing trigger event" in the logs | 03:03 |
| Thelo | jhesketh : This is the patchset I made this morning : zuul.Scheduler: Processing trigger event <TriggerEvent patchset-created openstack-dev/sandbox master 266638,1> | 03:05 |
| jhesketh | Thelo: okay any clues around that log line | 03:05 |
| openstackgerrit | Merged openstack/requirements: Adds networking-hyperv to projects.txt https://review.openstack.org/258987 | 03:07 |
| Thelo | jhesketh: http://paste.openstack.org/show/483681/ | 03:07 |
| *** yamamoto_ has joined #openstack-infra | 03:07 | |
| Thelo | that line : ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information. | 03:07 |
| jhesketh | Thelo: that event is for openstack-dev/sandbox, your layout.yaml is configured for the project openstack-dev/ci-sandbox | 03:08 |
| jhesketh | Thelo: so you'll probably want to update your layout | 03:08 |
| jhesketh | Thelo: the EventConnector issue doesn't matter | 03:08 |
| *** mtanino has joined #openstack-infra | 03:08 | |
| Thelo | Ooo | 03:08 |
| Thelo | jhesketh: very true | 03:09 |
| *** gildub has joined #openstack-infra | 03:10 | |
| *** thorst has joined #openstack-infra | 03:12 | |
| *** markmcclain has quit IRC | 03:12 | |
| Thelo | jhesketh, mmedvede: Many thanks I will try again | 03:13 |
| jhesketh | no worries :-) | 03:13 |
| *** kushal has joined #openstack-infra | 03:13 | |
| *** markmcclain has joined #openstack-infra | 03:15 | |
| *** apoorvad has quit IRC | 03:16 | |
| *** otter768 has joined #openstack-infra | 03:17 | |
| *** yamahata has quit IRC | 03:17 | |
| *** dims has joined #openstack-infra | 03:17 | |
| *** thorst has quit IRC | 03:19 | |
| *** ekarlso has joined #openstack-infra | 03:19 | |
| Thelo | jhesketh: I get the same result. | 03:19 |
| jhesketh | Thelo: got some new logs please? | 03:19 |
| *** yuanying has quit IRC | 03:19 | |
| *** mriedem has quit IRC | 03:20 | |
| *** weshay_xchat has quit IRC | 03:21 | |
| *** unicell has quit IRC | 03:21 | |
| jhesketh | Thelo: did you restart zuul after you updated your layout.yaml | 03:22 |
| jhesketh | or reconfigure it | 03:22 |
| Thelo | http://paste.openstack.org/show/483683/ | 03:22 |
| Thelo | I didn't updated layout.yaml | 03:22 |
| *** binbincong has joined #openstack-infra | 03:22 | |
| Thelo | I just did another commit on the right branch | 03:22 |
| Thelo | but when I try to stop zuul I get this error message : No process in pidfile '/var/run/zuul/zuul.pid' found running; none killed. | 03:23 |
| *** links has joined #openstack-infra | 03:23 | |
| *** rkukura has quit IRC | 03:24 | |
| *** claudiub has quit IRC | 03:24 | |
| dougwig | Thelo: do you see the pipelines initialized in /var/log/zuul/zuul.log ? Does 'service zuul stop' actually show it stopped in 'ps auxw | grep zuul' ? If not, you might be running afoul of the scheduler lock bug I was having last week. | 03:25 |
| jhesketh | Thelo: you may have to kill the process and make sure the pid file is writeable by the daemon user | 03:25 |
| jhesketh | file/path that is | 03:25 |
| jhesketh | Thelo: once you've got it running again, do a reload so it prints out the pipeline configuration and paste-bin that if you can please | 03:26 |
| *** edmondsw has quit IRC | 03:27 | |
| Thelo | dougwig : I dont see any initialization log | 03:28 |
| Thelo | dougwig: the zuul process are still there, can I just kill them and restart? | 03:29 |
| dougwig | you'll have to kill -9 them. then one sec, i have something for you to try. | 03:29 |
| dougwig | i'd suggest trying this, if it's hung on you repeated, and you don't see your pipelines initializing | 03:31 |
| dougwig | https://www.irccloud.com/pastebin/00tjSkvr/ | 03:31 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:32 | |
| Thelo | in zuul.log I only see this 2016-01-13 11:30:29,344 INFO zuul.Server: Starting scheduler | 03:32 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Set BRANCH for periodic unittest bitrot jobs https://review.openstack.org/245525 | 03:32 |
| *** dims has quit IRC | 03:33 | |
| *** yuanying has joined #openstack-infra | 03:33 | |
| *** gyee has quit IRC | 03:34 | |
| dougwig | Thelo: after you tried the above? i'd expect that before, if you hit the same issue that I did. | 03:34 |
| jhesketh | dougwig: why are you undoing the apscheduler bump? | 03:35 |
| *** gongysh has joined #openstack-infra | 03:36 | |
| dougwig | jhesketh: because that was the cause of my zuul hang. | 03:36 |
| Thelo | dougwig I am trying | 03:36 |
| *** vivekd has joined #openstack-infra | 03:36 | |
| *** woodster_ has quit IRC | 03:36 | |
| *** rlandy has quit IRC | 03:36 | |
| jhesketh | dougwig: do you have any more details about the zuul hang? | 03:36 |
| jhesketh | ie we should probably fix it if it's wrong | 03:37 |
| dougwig | jhesketh: yes, one sec. phone just rang | 03:37 |
| *** coolsvap|away is now known as coolsvap | 03:37 | |
| *** aeng_ has quit IRC | 03:37 | |
| *** markmcclain has quit IRC | 03:37 | |
| Thelo | ok now I see things http://paste.openstack.org/show/483686/ | 03:38 |
| *** Daisy has quit IRC | 03:38 | |
| *** markmcclain has joined #openstack-infra | 03:38 | |
| dougwig | Thelo: ok, yikes, you had the same bug. jhesketh, standby. | 03:40 |
| *** annegent_ has quit IRC | 03:40 | |
| dougwig | Thelo: you can continue with your CI with that revision of zuul, btw. | 03:41 |
| Thelo | dougwig: http://paste.openstack.org/show/483687/ I just ran into more issues | 03:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 03:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 03:44 |
| jhesketh | that LOG PATH can be covered by https://review.openstack.org/#/c/260243/1 | 03:45 |
| jhesketh | you'll need to check your success/failure patterns in the layout.yaml | 03:46 |
| *** unicell has joined #openstack-infra | 03:46 | |
| Thelo | http://paste.openstack.org/show/483688/ | 03:47 |
| Thelo | that one is more explicit | 03:47 |
| jhesketh | Thelo: line 46 looks promising there | 03:48 |
| Thelo | that does not look normal : zuul.Scheduler: Project None not found | 03:50 |
| *** unicell1 has joined #openstack-infra | 03:50 | |
| *** unicell has quit IRC | 03:50 | |
| *** yuanying_ has joined #openstack-infra | 03:52 | |
| *** yuanying has quit IRC | 03:52 | |
| *** zhurong has quit IRC | 03:52 | |
| *** markmcclain has quit IRC | 03:52 | |
| mmedvede | Thelo: that is normal in a sense, there is no project associated with ref-replicated event, ignore it | 03:53 |
| Thelo | mmedvede | 03:53 |
| Thelo | mmedvede: ok thanks | 03:54 |
| *** zhurong has joined #openstack-infra | 03:56 | |
| *** annegentle has joined #openstack-infra | 04:00 | |
| Thelo | jhesketh: http://paste.openstack.org/show/483689/ INFO zuul.IndependentPipelineManager: Unable to merge change <Change 0x7fa720079850 266657,3> | 04:03 |
| Thelo | jhesketh: I see some error in zull-merger now | 04:04 |
| *** Daisy has joined #openstack-infra | 04:04 | |
| jhesketh | Thelo: so possibly need to look at zuul-merger | 04:04 |
| jhesketh | yeah | 04:04 |
| jhesketh | are your /var/lib/zuul's owned correctly | 04:04 |
| Thelo | paste.openstack.org/show/483690/ | 04:04 |
| *** otter768 has quit IRC | 04:04 | |
| Thelo | who should be the owner? and what should be the right permission | 04:05 |
| *** yuanying_ has quit IRC | 04:05 | |
| Thelo | they should all belong to zuul right? | 04:05 |
| *** yuanying has joined #openstack-infra | 04:06 | |
| Thelo | In my case replacing zuul as mentioned by dougwig might have change some item to root owner | 04:06 |
| dougwig | jhesketh: ping, sorry for disappearing. | 04:07 |
| dougwig | Thelo: shouldn't have, unless you manually started zuul by hand. | 04:07 |
| *** yuanying_ has joined #openstack-infra | 04:07 | |
| jhesketh | Thelo: looks like you don't have the merger ssh key set up correctly for cloning from gerrit | 04:07 |
| jhesketh | Thelo: see line 32 of paste | 04:07 |
| dougwig | Thelo: you can ignore your ref-replicated errors. | 04:07 |
| dougwig | the traceback is a problem with your openstack_functions.py file, usually. | 04:08 |
| *** Daisy_ has joined #openstack-infra | 04:08 | |
| dougwig | jhesketh: I mentioned the zuul issue last week, though it was just me. my zuul started hanging right after the apscheduler bump, trying to launch a thread for the timer trigger. i'm not sure if it was an incompatible mix of package versions or what, but the apscheduler bump absolutely hosed my zuul. | 04:08 |
| dougwig | /though/thought | 04:09 |
| *** bpokorny has joined #openstack-infra | 04:09 | |
| jhesketh | hmm okay | 04:09 |
| jhesketh | dougwig: do you have logs? | 04:09 |
| *** tongli has joined #openstack-infra | 04:10 | |
| dougwig | merger logs were fine. the zuul logs had one line "Starting scheduler...", and *nothing* else. I actually put LOG.debug's into the zuul py files to find that it was bombing while trying to import the timer trigger on startup. | 04:10 |
| *** yuanying has quit IRC | 04:10 | |
| dougwig | and nothing but a kill -9 would kill zuul-server. it was wedged up tight. | 04:10 |
| *** salv-orl_ has joined #openstack-infra | 04:10 | |
| *** ccrouch has quit IRC | 04:11 | |
| Thelo | jhesketh: I dont see anything line 32 | 04:11 |
| *** ccrouch has joined #openstack-infra | 04:11 | |
| jhesketh | Thelo: "Host key verification failed." | 04:11 |
| jhesketh | Thelo: you need to add gerrit to your zuul user's known_hosts file | 04:11 |
| jhesketh | The host keys are here: https://review.openstack.org/#/settings/ssh-keys | 04:12 |
| *** Daisy has quit IRC | 04:12 | |
| *** Daisy_ is now known as Daisy | 04:12 | |
| dougwig | jhesketh: there's only one in the current thirdparty ci puppet, so i'm guessing that's the missing key. | 04:13 |
| *** salv-orlando has quit IRC | 04:13 | |
| *** kushal has quit IRC | 04:13 | |
| *** tzn has joined #openstack-infra | 04:16 | |
| *** thorst has joined #openstack-infra | 04:17 | |
| Thelo | jhesketh: Thanks I couldn't find the log you mentioned but after adding the key it seems to work | 04:19 |
| *** kushal has joined #openstack-infra | 04:20 | |
| *** tzn has quit IRC | 04:20 | |
| Thelo | one more question, what is this noop-check-communication? | 04:21 |
| *** tongli has quit IRC | 04:23 | |
| *** ccrouch has quit IRC | 04:24 | |
| *** markmcclain has joined #openstack-infra | 04:24 | |
| *** tongli has joined #openstack-infra | 04:24 | |
| *** thorst has quit IRC | 04:25 | |
| dougwig | Thelo: that just a starter job that does nothing but echo hello world, for testing that your zuul/gearman/jenkins and all are working properly. | 04:25 |
| *** vivekd_ has joined #openstack-infra | 04:25 | |
| dougwig | /that/that's/ | 04:25 |
| *** tongli has quit IRC | 04:26 | |
| Thelo | dougwig: THanks does it report something to gerrit? | 04:26 |
| *** tongli has joined #openstack-infra | 04:26 | |
| dougwig | if it's in silent in zuul layout.yaml, no. if it's in check, yes. | 04:26 |
| Thelo | mmm in my case it didn't report anything | 04:27 |
| Thelo | however dsvm-tempest-full was trigger in jenkins | 04:27 |
| Thelo | so there is something wrong right? | 04:27 |
| *** bharathm has joined #openstack-infra | 04:27 | |
| *** vivekd has quit IRC | 04:27 | |
| *** vivekd_ is now known as vivekd | 04:27 | |
| dougwig | i had issues with that job sitting in jenkins queue and not running on master like it was supposed to. i moved it to the dpc nodes and it started running fine. i think there's a setup bug in that part. | 04:28 |
| *** markmcclain has quit IRC | 04:29 | |
| *** Daisy has quit IRC | 04:30 | |
| *** tongli has quit IRC | 04:30 | |
| jhesketh | Thelo: now all the problems are solved, do you want to try upgrading to zuul head and seeing if it still works | 04:30 |
| jhesketh | would be helpful to see if you've got the same problem as dougwig if you don't mind | 04:30 |
| openstackgerrit | sonu proposed openstack-infra/project-config: Adds the Bandit job for Designate in OpenStack Infra https://review.openstack.org/266671 | 04:31 |
| dougwig | jhesketh, Thelo: ooh yes, that would be a useful test. | 04:31 |
| *** tongli has joined #openstack-infra | 04:31 | |
| Thelo | haha | 04:31 |
| Thelo | ok | 04:31 |
| Thelo | just tell me how to do | 04:31 |
| *** bharathm has quit IRC | 04:31 | |
| dougwig | Thelo: one sec | 04:33 |
| dougwig | https://www.irccloud.com/pastebin/rnrD7IJz/ | 04:33 |
| Thelo | dougwig shouldn't I uninstall zuul before installing it again? | 04:35 |
| *** tongli has quit IRC | 04:35 | |
| dougwig | the upgrade would be better, because you might keep some min rev packages. you can always try fresh after if it works somehow. | 04:36 |
| *** tongli has joined #openstack-infra | 04:36 | |
| *** treaki_ has joined #openstack-infra | 04:37 | |
| Thelo | dougwig: not sure that things were actually upgraded : http://paste.openstack.org/show/483692/ | 04:38 |
| dougwig | ok, then "pip uninstall zuul ; pip install zuul". | 04:39 |
| dougwig | you'll know you have it right when "pip list | grep apsched" returns a version of >3.x | 04:39 |
| *** vivekd_ has joined #openstack-infra | 04:39 | |
| *** tongli has quit IRC | 04:40 | |
| *** dimtruck is now known as zz_dimtruck | 04:40 | |
| *** vivekd has quit IRC | 04:41 | |
| *** vivekd_ is now known as vivekd | 04:41 | |
| Thelo | piplist | grep apsch returns nothing. | 04:41 |
| *** treaki__ has quit IRC | 04:41 | |
| dougwig | try this: | 04:43 |
| dougwig | python -c "import apscheduler; print apscheduler.__version__" | 04:43 |
| Thelo | AttributeError: 'module' object has no attribute '__version__' | 04:43 |
| dougwig | that's the old one. | 04:44 |
| *** zhurong has quit IRC | 04:44 | |
| dougwig | try: | 04:44 |
| dougwig | pip install -U git+https://github.com/openstack-infra/zuul.git | 04:45 |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 pypy check for kolla project https://review.openstack.org/266674 | 04:45 |
| *** zhurong has joined #openstack-infra | 04:46 | |
| *** sdake_ has joined #openstack-infra | 04:47 | |
| *** sdake has quit IRC | 04:48 | |
| Thelo | dougwig: I ok I have the right version now | 04:48 |
| Thelo | I am triggering a new commit | 04:48 |
| dougwig | Thelo: great. fire up that zuul and let's see what happens. | 04:48 |
| zaro | jhesketh: take another look? https://review.openstack.org/#/c/261110/ | 04:48 |
| Thelo | it seems ok | 04:49 |
| Thelo | dougwig: For this commit I should trigger both noop-check-communication and dsvm tempest-fuul, in jenkins it looks like it proccess them one by one | 04:50 |
| Thelo | why not in parrallel> | 04:50 |
| Thelo | ? | 04:50 |
| dougwig | if you see zuul pipelines, it's working. drat that it didn't repro. | 04:50 |
| dougwig | do you have enough jobs available to run both at the same time? | 04:51 |
| Thelo | I have 4 slaves, all of them are up but only one of them is working | 04:51 |
| *** sdake_ has quit IRC | 04:53 | |
| Thelo | O I see it is node affected to any nodes type | 04:54 |
| Thelo | is it normal? | 04:55 |
| *** sdake has joined #openstack-infra | 04:55 | |
| *** hdd has joined #openstack-infra | 04:56 | |
| *** Daisy has joined #openstack-infra | 04:56 | |
| *** zhurong has quit IRC | 04:59 | |
| *** jamespage has quit IRC | 05:00 | |
| *** gnuoy has quit IRC | 05:00 | |
| *** gnuoy has joined #openstack-infra | 05:00 | |
| jhesketh | zaro: looking | 05:02 |
| *** Sukhdev has joined #openstack-infra | 05:03 | |
| *** zhurong has joined #openstack-infra | 05:03 | |
| *** boris-42 has quit IRC | 05:03 | |
| *** piet has joined #openstack-infra | 05:04 | |
| dougwig | Thelo: if your node doesn't match your job, it's normal to not run. that's all about your config. | 05:06 |
| *** Swami has joined #openstack-infra | 05:08 | |
| openstackgerrit | Qiming Teng proposed openstack-infra/irc-meetings: Fix broken ptl list tool https://review.openstack.org/266683 | 05:12 |
| *** markmcclain has joined #openstack-infra | 05:12 | |
| *** sdake has quit IRC | 05:16 | |
| *** tzn has joined #openstack-infra | 05:17 | |
| *** sridhar_ram has joined #openstack-infra | 05:18 | |
| *** jamespage has joined #openstack-infra | 05:19 | |
| *** markmcclain has quit IRC | 05:21 | |
| *** thorst has joined #openstack-infra | 05:21 | |
| Thelo | dougwig: Thanks | 05:22 |
| *** tzn has quit IRC | 05:22 | |
| *** thorst has quit IRC | 05:29 | |
| *** annegentle has quit IRC | 05:29 | |
| *** mrmartin has joined #openstack-infra | 05:29 | |
| *** Qiming_ has joined #openstack-infra | 05:30 | |
| openstackgerrit | sonu proposed openstack-infra/project-config: Adds the Bandit job for Designate in OpenStack Infra https://review.openstack.org/266671 | 05:33 |
| *** Qiming has quit IRC | 05:33 | |
| *** piet has quit IRC | 05:38 | |
| *** jaosorior has joined #openstack-infra | 05:41 | |
| *** mtanino has quit IRC | 05:43 | |
| *** _nadya_ has joined #openstack-infra | 05:48 | |
| *** markmcclain has joined #openstack-infra | 05:50 | |
| *** bpokorny has quit IRC | 05:53 | |
| *** markmcclain has quit IRC | 05:56 | |
| *** mrmartin has quit IRC | 06:01 | |
| *** mrmartin has joined #openstack-infra | 06:03 | |
| *** mrmartin has quit IRC | 06:08 | |
| *** vgridnev has joined #openstack-infra | 06:13 | |
| *** _nadya_ has quit IRC | 06:14 | |
| *** tzn has joined #openstack-infra | 06:18 | |
| *** ParsectiX has quit IRC | 06:22 | |
| *** tzn has quit IRC | 06:22 | |
| *** vivekd has quit IRC | 06:23 | |
| *** gongysh has quit IRC | 06:24 | |
| *** thorst has joined #openstack-infra | 06:26 | |
| *** markvoelker has quit IRC | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/265697 | 06:30 |
| *** ParsectiX has joined #openstack-infra | 06:33 | |
| *** thorst has quit IRC | 06:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:34 | |
| *** amit213 has quit IRC | 06:35 | |
| *** hdd has quit IRC | 06:35 | |
| *** amit213 has joined #openstack-infra | 06:36 | |
| openstackgerrit | sonu proposed openstack-infra/project-config: Add the Bandit gate to Designate section in OpenStack Infra https://review.openstack.org/266721 | 06:38 |
| *** vivekd has joined #openstack-infra | 06:40 | |
| *** yamahata has joined #openstack-infra | 06:48 | |
| *** mrunge has joined #openstack-infra | 06:48 | |
| *** zhurong has quit IRC | 06:49 | |
| *** rossella_s has quit IRC | 06:52 | |
| *** Sukhdev has quit IRC | 06:52 | |
| *** rossella_s has joined #openstack-infra | 06:52 | |
| *** jaosorior has quit IRC | 06:57 | |
| *** aopincar_ has joined #openstack-infra | 06:58 | |
| *** sridhar_ram has quit IRC | 07:02 | |
| *** sridhar_ram has joined #openstack-infra | 07:06 | |
| *** Swami has quit IRC | 07:10 | |
| *** mudassirlatif has joined #openstack-infra | 07:12 | |
| *** med_ has quit IRC | 07:12 | |
| *** med_ has joined #openstack-infra | 07:14 | |
| *** med_ is now known as Guest77121 | 07:14 | |
| *** zhurong has joined #openstack-infra | 07:15 | |
| *** vivekd has quit IRC | 07:15 | |
| *** isaacb has joined #openstack-infra | 07:17 | |
| *** tzn has joined #openstack-infra | 07:19 | |
| *** gildub has quit IRC | 07:19 | |
| *** zhurong has quit IRC | 07:20 | |
| *** zhurong has joined #openstack-infra | 07:23 | |
| *** tzn has quit IRC | 07:23 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 pypy check for kolla project https://review.openstack.org/266674 | 07:24 |
| *** markvoelker has joined #openstack-infra | 07:28 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 check for kolla project https://review.openstack.org/266674 | 07:28 |
| *** jtomasek has joined #openstack-infra | 07:28 | |
| *** thorst has joined #openstack-infra | 07:31 | |
| *** markvoelker has quit IRC | 07:33 | |
| *** cnesa has joined #openstack-infra | 07:33 | |
| *** thorst has quit IRC | 07:39 | |
| *** _nadya_ has joined #openstack-infra | 07:39 | |
| *** e0ne has joined #openstack-infra | 07:41 | |
| *** kzaitsev_mb has quit IRC | 07:41 | |
| *** vgridnev has quit IRC | 07:42 | |
| AJaeger | There's a job that has been approved which is not entering the gate - even after several rechecks: https://review.openstack.org/#/c/265982 Could an infra-root investigate, please? | 07:45 |
| *** tzn has joined #openstack-infra | 07:47 | |
| yolanda | hi AJaeger, morning | 07:50 |
| yolanda | i can take a look | 07:50 |
| AJaeger | morning, yolanda ! | 07:50 |
| AJaeger | thanks | 07:51 |
| *** YorikSar has joined #openstack-infra | 07:52 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project https://review.openstack.org/266674 | 07:52 |
| yolanda | AJaeger, so it will need another +1 from someone. recheck will make it go to check queue, but not to gate. It needs an extra +w from another reviewer, or -w +w from dims | 07:54 |
| YorikSar | Oh, it looks like eavesdrop lags a bit here. | 07:56 |
| YorikSar | yolanda: I just added +W there and it got to Gate queue. | 07:56 |
| yolanda | cool | 07:57 |
| YorikSar | (I assume you're talking about https://review.openstack.org/265982) | 07:57 |
| *** mpavone has joined #openstack-infra | 07:57 | |
| AJaeger | yolanda: but why? Looking at history joshua harlow gave +2, then dims a +W, then dims a +2 | 07:57 |
| AJaeger | YorikSar: yes | 07:57 |
| AJaeger | So, it should have gone to gate already after dims' +W | 07:57 |
| yolanda | AJaeger, maybe there was some restart, or some unexpected problem that caused that? | 07:58 |
| YorikSar | My guess is that Zuul missed the original Verified+1 from Jenkins and it didn't trigger smth in Zuul. | 07:58 |
| yolanda | yep, that is normally happening when for some reason gerrit fails to send the event, or zuul to pick it up | 07:58 |
| *** ifarkas has joined #openstack-infra | 07:59 | |
| YorikSar | Last reconfigured: Tue Jan 12 2016 14:36:48 GMT+0300 | 08:00 |
| openstackgerrit | Merged openstack-infra/project-config: Fixed mistral experimental job https://review.openstack.org/265030 | 08:00 |
| YorikSar | Maybe Zuul forgot about some events after restart? | 08:00 |
| *** otter768 has joined #openstack-infra | 08:00 | |
| AJaeger | YorikSar: reconfigured is just reload, restart is different - and that one happened as well this week. | 08:00 |
| *** ildikov has quit IRC | 08:00 | |
| AJaeger | yolanda: might be. Ok, case solved for now - thanks! | 08:00 |
| *** otter768 has quit IRC | 08:02 | |
| YorikSar | yolanda: Do you know if Zuul periodically checks if it missed something from event stream? | 08:02 |
| *** otter768 has joined #openstack-infra | 08:02 | |
| yolanda | no, it doesn't | 08:02 |
| YorikSar | I wonder if it would be a good idea to query Gerrit once in a while (an hour may be) for changes that should be in queues an verify that those are in queues. | 08:03 |
| *** salv-orl_ has quit IRC | 08:04 | |
| *** otter768 has quit IRC | 08:04 | |
| *** salv-orlando has joined #openstack-infra | 08:04 | |
| yolanda | it's a recurring problem we have, but we are solving manually. It seems a useful idea to me, but i'm not sure about the complexity | 08:04 |
| *** otter768_ has joined #openstack-infra | 08:04 | |
| AJaeger | YorikSar: this is a corner case and worth doing after a restart I guess - better discuss with the Zuul experts at US time. | 08:04 |
| yolanda | YorikSar, you can throw that to the mailing list | 08:04 |
| *** vgridnev has joined #openstack-infra | 08:04 | |
| *** Thelo has quit IRC | 08:04 | |
| AJaeger | thanks for the suggestions, YorikSar | 08:06 |
| *** otter768_ has quit IRC | 08:06 | |
| *** otter768 has joined #openstack-infra | 08:06 | |
| *** jistr has joined #openstack-infra | 08:07 | |
| *** tzn has quit IRC | 08:07 | |
| *** Thelo has joined #openstack-infra | 08:07 | |
| *** jistr is now known as jistr|doc | 08:07 | |
| *** otter768 has quit IRC | 08:08 | |
| *** otter768_ has joined #openstack-infra | 08:08 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files https://review.openstack.org/243399 | 08:08 |
| YorikSar | Ok, I'll write it up to ML later today. | 08:08 |
| *** otter768 has joined #openstack-infra | 08:10 | |
| *** otter768_ has quit IRC | 08:10 | |
| *** Hal has joined #openstack-infra | 08:11 | |
| *** Hal is now known as Guest22649 | 08:11 | |
| *** flepied1 has quit IRC | 08:11 | |
| *** otter768_ has joined #openstack-infra | 08:12 | |
| openstackgerrit | Merged openstack/requirements: ryu 3.29 incompatible with oslo.config https://review.openstack.org/264741 | 08:12 |
| *** otter768_ has quit IRC | 08:13 | |
| *** markus_z has joined #openstack-infra | 08:13 | |
| *** otter768_ has joined #openstack-infra | 08:13 | |
| *** otter768 has quit IRC | 08:15 | |
| *** abregman has joined #openstack-infra | 08:15 | |
| *** otter768 has joined #openstack-infra | 08:15 | |
| *** otter768_ has quit IRC | 08:16 | |
| Thelo | jhesketh, dougwig : are you still here? | 08:17 |
| jhesketh | Thelo: in and out, but yes | 08:17 |
| Thelo | zuul stops at Starting scheduler again | 08:17 |
| *** otter768_ has joined #openstack-infra | 08:17 | |
| *** otter768 has quit IRC | 08:17 | |
| jhesketh | Thelo: oh that's interesting... so you might be seeing what dougwig was | 08:18 |
| jhesketh | Thelo: can you post your logs somewhere per chance? | 08:18 |
| Thelo | sure | 08:18 |
| *** e0ne has quit IRC | 08:18 | |
| *** otter768 has joined #openstack-infra | 08:19 | |
| *** otter768_ has quit IRC | 08:19 | |
| Thelo | jhesketh: http://paste.openstack.org/show/483703/ | 08:19 |
| jhesketh | thanks | 08:20 |
| *** otter768_ has joined #openstack-infra | 08:21 | |
| *** otter768 has quit IRC | 08:21 | |
| jhesketh | Thelo: could you do a pip freeze please? | 08:22 |
| *** otter768_ has quit IRC | 08:23 | |
| *** otter768 has joined #openstack-infra | 08:23 | |
| Thelo | jhesketh: paste.openstack.org/show/483707/ however I have already replace zuul | 08:23 |
| jhesketh | Thelo: you reverted to before the APScheduler update? | 08:24 |
| *** shardy has joined #openstack-infra | 08:24 | |
| Thelo | jhesketh: not intentionnally at least | 08:25 |
| *** otter768 has quit IRC | 08:25 | |
| jhesketh | Thelo: what do you mean by replace zuul then? | 08:25 |
| *** otter768 has joined #openstack-infra | 08:25 | |
| *** vgridnev has quit IRC | 08:25 | |
| *** otter768_ has joined #openstack-infra | 08:27 | |
| *** HeOS has quit IRC | 08:27 | |
| *** dtardivel has joined #openstack-infra | 08:28 | |
| *** otter76__ has joined #openstack-infra | 08:29 | |
| *** otter768 has quit IRC | 08:29 | |
| *** otter768_ has quit IRC | 08:29 | |
| Thelo | jhesketh: Even when I do doug's method it still stuck at the same point | 08:29 |
| *** matrohon has joined #openstack-infra | 08:29 | |
| *** ildikov has joined #openstack-infra | 08:29 | |
| *** mudassirlatif has quit IRC | 08:29 | |
| jhesketh | hmm okay | 08:29 |
| openstackgerrit | yolanda.robla proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files https://review.openstack.org/243399 | 08:29 |
| jhesketh | Thelo: are you looking at the debug log, or just zuul.log? | 08:30 |
| jhesketh | Thelo: I assume you've restarted zuul and cleared out /var/lib/zuul etc | 08:30 |
| Thelo | jhesketh: What should I clean in /var/lib/zuul? | 08:31 |
| *** otter768 has joined #openstack-infra | 08:31 | |
| *** otter76__ has quit IRC | 08:31 | |
| jhesketh | when restarting it's just useful to clear out any state that zuul might have stored | 08:31 |
| jhesketh | you should be able to clear all of it out reasonably safely | 08:31 |
| *** yaume has joined #openstack-infra | 08:32 | |
| *** yamahata has quit IRC | 08:32 | |
| *** otter768 has quit IRC | 08:32 | |
| *** samuelBartel has joined #openstack-infra | 08:32 | |
| *** otter768 has joined #openstack-infra | 08:33 | |
| *** GheRiver1 has joined #openstack-infra | 08:34 | |
| yolanda | AJaeger, do you know if there is some problem with ansible-lint checks | 08:34 |
| yolanda | ? | 08:34 |
| yolanda | i see my tests failing with 2016-01-13 08:32:50.448 | AttributeError: 'module' object has no attribute 'parse_yaml_from_file' | 08:34 |
| *** otter768_ has joined #openstack-infra | 08:34 | |
| *** GheRiver1 has quit IRC | 08:35 | |
| *** esikachev has joined #openstack-infra | 08:36 | |
| *** otter768_ has quit IRC | 08:36 | |
| *** otter76__ has joined #openstack-infra | 08:36 | |
| AJaeger | yolanda: No idea, do you have a change to look at? | 08:37 |
| *** arxcruz has joined #openstack-infra | 08:37 | |
| yolanda | https://review.openstack.org/243399 | 08:37 |
| *** thorst has joined #openstack-infra | 08:37 | |
| *** otter768 has quit IRC | 08:37 | |
| *** kushal has quit IRC | 08:37 | |
| AJaeger | yolanda: ansible 2.0 was released yesterday, might be a fall out from that | 08:38 |
| *** otter768 has joined #openstack-infra | 08:38 | |
| *** otter76__ has quit IRC | 08:38 | |
| yolanda | it may be a problem with the test itself, i'm running using ansible 2.0 and works fine | 08:39 |
| *** nmagnezi has joined #openstack-infra | 08:39 | |
| AJaeger | can't help with that - perhaps ask pabelanger | 08:39 |
| yolanda | yep, i'll try find some time to take a look, or ping him later | 08:39 |
| *** shardy has quit IRC | 08:40 | |
| *** otter768 has quit IRC | 08:40 | |
| *** otter768_ has joined #openstack-infra | 08:40 | |
| *** daemontool has joined #openstack-infra | 08:40 | |
| *** flepied has joined #openstack-infra | 08:40 | |
| *** fhubik has joined #openstack-infra | 08:41 | |
| *** otter768_ has quit IRC | 08:42 | |
| *** dingyichen has quit IRC | 08:42 | |
| *** shardy has joined #openstack-infra | 08:42 | |
| *** otter768 has joined #openstack-infra | 08:42 | |
| *** thorst has quit IRC | 08:43 | |
| *** ihrachys has joined #openstack-infra | 08:43 | |
| *** daemontool_ has joined #openstack-infra | 08:43 | |
| *** daemontool has quit IRC | 08:43 | |
| *** otter768_ has joined #openstack-infra | 08:44 | |
| *** daemontool_ is now known as daemontool | 08:44 | |
| *** otter768 has quit IRC | 08:44 | |
| *** jlanoux has joined #openstack-infra | 08:45 | |
| *** otter768 has joined #openstack-infra | 08:46 | |
| openstackgerrit | IWAMOTO Toshihiro proposed openstack/requirements: Bump ryu to 3.29.1 https://review.openstack.org/264026 | 08:46 |
| *** otter768_ has quit IRC | 08:46 | |
| *** otter768 has quit IRC | 08:48 | |
| *** otter768_ has joined #openstack-infra | 08:48 | |
| *** kushal has joined #openstack-infra | 08:49 | |
| *** otter768_ has quit IRC | 08:50 | |
| *** otter768 has joined #openstack-infra | 08:50 | |
| *** otter768_ has joined #openstack-infra | 08:52 | |
| *** otter768 has quit IRC | 08:52 | |
| Thelo | jhesketh: actually I just modified the layout.yaml then it stoped working, after I rolled back it seems fine... | 08:52 |
| Thelo | jhesketh: do you see anything wrong in the commit before the last one (https://github.com/thelo-gaultier/itri-disco-ci/blob/master/zuul/layout.yaml)? | 08:54 |
| *** otter768 has joined #openstack-infra | 08:54 | |
| *** markvoelker has joined #openstack-infra | 08:54 | |
| *** otter768 has quit IRC | 08:55 | |
| *** otter76__ has joined #openstack-infra | 08:55 | |
| *** otter768_ has quit IRC | 08:56 | |
| *** otter768 has joined #openstack-infra | 08:57 | |
| *** sshnaidm has quit IRC | 08:59 | |
| *** markvoelker has quit IRC | 08:59 | |
| *** otter768_ has joined #openstack-infra | 08:59 | |
| *** otter768 has quit IRC | 08:59 | |
| *** otter76__ has quit IRC | 09:00 | |
| *** otter768_ has quit IRC | 09:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:01 | |
| *** salv-orlando has quit IRC | 09:02 | |
| *** MCoLo has joined #openstack-infra | 09:02 | |
| *** dizquierdo has joined #openstack-infra | 09:02 | |
| *** sfinucan has joined #openstack-infra | 09:02 | |
| *** salv-orlando has joined #openstack-infra | 09:03 | |
| *** gongysh has joined #openstack-infra | 09:03 | |
| *** otter768 has joined #openstack-infra | 09:03 | |
| *** otter768 has quit IRC | 09:05 | |
| *** otter768_ has joined #openstack-infra | 09:05 | |
| *** otter768 has joined #openstack-infra | 09:07 | |
| Thelo | jhesketh: problem fixed | 09:08 |
| *** otter768 has quit IRC | 09:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Update Watcher Tempest tests package name https://review.openstack.org/266430 | 09:09 |
| *** otter768 has joined #openstack-infra | 09:09 | |
| oomichi | hi, can somebody take alook at a cliff patch? https://review.openstack.org/#/c/266744/ | 09:10 |
| *** amotoki has joined #openstack-infra | 09:10 | |
| oomichi | the pep8 failure happens on the gate now, and the patch fixes it. | 09:10 |
| *** otter768_ has quit IRC | 09:10 | |
| openstackgerrit | Chris Jones proposed openstack-infra/project-config: Add project configuration for Vahana. https://review.openstack.org/266549 | 09:10 |
| *** zhurong has quit IRC | 09:11 | |
| *** GheRiver1 has joined #openstack-infra | 09:11 | |
| *** otter768_ has joined #openstack-infra | 09:11 | |
| *** otter768 has quit IRC | 09:11 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project https://review.openstack.org/266674 | 09:11 |
| *** yaume_ has joined #openstack-infra | 09:11 | |
| *** MCoLo has quit IRC | 09:12 | |
| *** MCoLo has joined #openstack-infra | 09:12 | |
| *** GheRiver1 has quit IRC | 09:12 | |
| *** otter768_ has quit IRC | 09:13 | |
| *** otter768 has joined #openstack-infra | 09:13 | |
| *** yaume has quit IRC | 09:14 | |
| *** otter768_ has joined #openstack-infra | 09:15 | |
| *** vivekd has joined #openstack-infra | 09:15 | |
| *** otter768 has quit IRC | 09:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add osc-plugin-jobs check job for all OSC plugins https://review.openstack.org/265608 | 09:15 |
| openstackgerrit | Merged openstack-infra/project-config: add ironic-inspector and tripleo to osc-check-plugins job https://review.openstack.org/265609 | 09:15 |
| *** otter768 has joined #openstack-infra | 09:17 | |
| AJaeger | oomichi: channel for cliff is #openstack-sdks - and probably also #openstack-oslo | 09:17 |
| openstackgerrit | Merged openstack-infra/project-config: Use file permissions instead of ownership to grant access to files https://review.openstack.org/266482 | 09:17 |
| *** jaosorior has joined #openstack-infra | 09:18 | |
| oomichi | AJaeger: thanks, nice info :) | 09:18 |
| *** otter768 has quit IRC | 09:19 | |
| *** otter76__ has joined #openstack-infra | 09:19 | |
| *** HeOS has joined #openstack-infra | 09:19 | |
| *** jlanoux has quit IRC | 09:19 | |
| *** otter768_ has quit IRC | 09:20 | |
| *** otter76__ has quit IRC | 09:20 | |
| *** jlanoux has joined #openstack-infra | 09:20 | |
| *** otter768 has joined #openstack-infra | 09:20 | |
| *** otter768 has quit IRC | 09:22 | |
| Thelo | Hi all, my jenkins slave tried to git fetch the code from my CI server instead of the openstack review server (http://paste.openstack.org/show/483715/) does anyone know why? | 09:22 |
| *** otter768 has joined #openstack-infra | 09:22 | |
| *** zeih has joined #openstack-infra | 09:24 | |
| *** otter768_ has joined #openstack-infra | 09:24 | |
| *** otter768 has quit IRC | 09:24 | |
| *** otter768_ has quit IRC | 09:26 | |
| *** otter768 has joined #openstack-infra | 09:26 | |
| *** AJaeger has quit IRC | 09:27 | |
| *** otter768 has quit IRC | 09:28 | |
| *** salv-orlando has quit IRC | 09:28 | |
| *** otter768 has joined #openstack-infra | 09:28 | |
| *** salv-orlando has joined #openstack-infra | 09:28 | |
| *** vgridnev has joined #openstack-infra | 09:29 | |
| *** otter768 has quit IRC | 09:30 | |
| *** otter768_ has joined #openstack-infra | 09:30 | |
| odyssey4me | yolanda yes, ansible-lint is not yet compatible with ansible 2.0 | 09:31 |
| odyssey4me | yolanda https://github.com/willthames/ansible-lint/issues/75 | 09:32 |
| *** otter768_ has quit IRC | 09:32 | |
| *** otter768 has joined #openstack-infra | 09:32 | |
| odyssey4me | yolanda for those of us using ansible-lint/ansible, we've currently capped ansible at <2.0 until ansible-lint has released a fix | 09:32 |
| *** e0ne has joined #openstack-infra | 09:33 | |
| *** otter768_ has joined #openstack-infra | 09:34 | |
| *** otter768 has quit IRC | 09:34 | |
| *** derekh has joined #openstack-infra | 09:35 | |
| *** dizquierdo has quit IRC | 09:36 | |
| *** otter768_ has quit IRC | 09:36 | |
| *** otter768 has joined #openstack-infra | 09:36 | |
| yolanda | odyssey4me, thanks, i'll try that fix | 09:37 |
| *** otter768_ has joined #openstack-infra | 09:38 | |
| *** tzn has joined #openstack-infra | 09:38 | |
| *** gongysh has quit IRC | 09:38 | |
| *** tzn has quit IRC | 09:39 | |
| *** otter76__ has joined #openstack-infra | 09:39 | |
| *** otter768_ has quit IRC | 09:40 | |
| *** otter768 has quit IRC | 09:40 | |
| *** Daisy has quit IRC | 09:40 | |
| *** thorst has joined #openstack-infra | 09:41 | |
| *** electrofelix has joined #openstack-infra | 09:41 | |
| *** otter76__ has quit IRC | 09:41 | |
| *** otter768 has joined #openstack-infra | 09:41 | |
| *** otter768 has quit IRC | 09:43 | |
| *** otter768_ has joined #openstack-infra | 09:44 | |
| *** otter768_ has quit IRC | 09:45 | |
| *** otter768 has joined #openstack-infra | 09:45 | |
| *** sshnaidm has joined #openstack-infra | 09:46 | |
| *** otter768 has quit IRC | 09:47 | |
| *** otter768 has joined #openstack-infra | 09:47 | |
| *** oomichi is now known as oomichi_away | 09:47 | |
| *** thorst has quit IRC | 09:48 | |
| Thelo | yolanda: Morning, in the openstack CI do you know where the variables "ZUUL_URL="http://hps10.ccma.itri.org.tw/p/"" is defined? | 09:49 |
| *** otter768_ has joined #openstack-infra | 09:49 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Fix broken ptl list tool https://review.openstack.org/266683 | 09:51 |
| *** otter76__ has joined #openstack-infra | 09:51 | |
| *** otter768_ has quit IRC | 09:51 | |
| yolanda | it is a parameter passed to zuul_scheduler or zuul_merger . For the merger it defaults to https://${::fqdn}/p, but you can override it | 09:51 |
| *** otter768 has quit IRC | 09:52 | |
| Thelo | yolanda, I tried to override it in zuul.conf unsuccessfully. You meant directly in the slave jobs? | 09:52 |
| *** otter768 has joined #openstack-infra | 09:53 | |
| *** otter76__ has quit IRC | 09:53 | |
| yolanda | i was talking about the puppet manifest itself | 09:54 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job https://review.openstack.org/260466 | 09:55 |
| *** otter768_ has joined #openstack-infra | 09:55 | |
| Thelo | yolanda: but right value should be review.openstack.org? why is the localhost the default value? | 09:55 |
| *** coolsvap is now known as coolsvap|away | 09:56 | |
| yolanda | Thelo, zuul_url is mostly used for the merger, to make it aware of the zuul server it needs to listen | 09:56 |
| *** otter768_ has quit IRC | 09:57 | |
| *** otter76__ has joined #openstack-infra | 09:57 | |
| yolanda | review.openstack.org is gerrit, not zuul | 09:57 |
| *** otter768 has quit IRC | 09:58 | |
| Thelo | yolanda: so why devstack try to get the code with this variable? | 09:58 |
| *** otter768 has joined #openstack-infra | 09:59 | |
| yolanda | it may need a different var than the one from zuul config, sorry. That depends on job configuration | 09:59 |
| *** otter768_ has joined #openstack-infra | 10:01 | |
| *** tzn has joined #openstack-infra | 10:01 | |
| Thelo | yolanda: you mean I should override this variable elsewhere? | 10:01 |
| *** dtantsur|afk is now known as dtantsur | 10:02 | |
| *** otter76__ has quit IRC | 10:02 | |
| *** otter768_ has quit IRC | 10:02 | |
| *** otter768_ has joined #openstack-infra | 10:03 | |
| yolanda | sorry, i cannot help you with devstack tests so much | 10:03 |
| yolanda | i thought you were talking about zuul configuration itself | 10:03 |
| *** otter768 has quit IRC | 10:04 | |
| *** otter768_ has quit IRC | 10:04 | |
| *** otter768 has joined #openstack-infra | 10:05 | |
| *** otter768_ has joined #openstack-infra | 10:06 | |
| *** otter768 has quit IRC | 10:06 | |
| *** otter768 has joined #openstack-infra | 10:08 | |
| *** GheRiver1 has joined #openstack-infra | 10:08 | |
| *** salv-orl_ has joined #openstack-infra | 10:09 | |
| *** otter76__ has joined #openstack-infra | 10:10 | |
| *** otter768 has quit IRC | 10:10 | |
| *** otter768_ has quit IRC | 10:11 | |
| *** GheRiver1 has quit IRC | 10:12 | |
| *** otter768 has joined #openstack-infra | 10:12 | |
| *** otter76__ has quit IRC | 10:12 | |
| *** GheRiver1 has joined #openstack-infra | 10:12 | |
| *** salv-orlando has quit IRC | 10:13 | |
| *** electrofelix has quit IRC | 10:13 | |
| jhesketh | Thelo: sorry was away.. so it was just a bad layout.yaml that stopped the scheduler from loading? | 10:13 |
| *** otter768_ has joined #openstack-infra | 10:14 | |
| *** hashar has joined #openstack-infra | 10:15 | |
| *** cdent has joined #openstack-infra | 10:15 | |
| *** otter76__ has joined #openstack-infra | 10:16 | |
| *** otter768_ has quit IRC | 10:16 | |
| *** hashar has quit IRC | 10:16 | |
| *** salv-orl_ has quit IRC | 10:16 | |
| *** otter768 has quit IRC | 10:17 | |
| *** GheRiver1 has quit IRC | 10:17 | |
| *** salv-orlando has joined #openstack-infra | 10:17 | |
| *** GheRiver1 has joined #openstack-infra | 10:17 | |
| *** otter768 has joined #openstack-infra | 10:18 | |
| *** rguillebert has joined #openstack-infra | 10:19 | |
| *** Qiming_ has quit IRC | 10:19 | |
| *** otter768 has quit IRC | 10:19 | |
| *** jistr|doc is now known as jistr | 10:19 | |
| *** otter768 has joined #openstack-infra | 10:20 | |
| *** nilasae has joined #openstack-infra | 10:20 | |
| *** electrofelix has joined #openstack-infra | 10:20 | |
| *** otter76__ has quit IRC | 10:21 | |
| *** otter768_ has joined #openstack-infra | 10:22 | |
| *** otter768 has quit IRC | 10:22 | |
| *** vivekd_ has joined #openstack-infra | 10:22 | |
| *** vivekd has quit IRC | 10:23 | |
| *** vivekd_ is now known as vivekd | 10:23 | |
| *** otter768 has joined #openstack-infra | 10:23 | |
| *** otter768_ has quit IRC | 10:24 | |
| *** otter768 has quit IRC | 10:25 | |
| *** otter768 has joined #openstack-infra | 10:25 | |
| *** jistr has quit IRC | 10:26 | |
| *** otter768_ has joined #openstack-infra | 10:27 | |
| *** otter768 has quit IRC | 10:27 | |
| *** otter768 has joined #openstack-infra | 10:29 | |
| *** otter768_ has quit IRC | 10:29 | |
| *** gildub has joined #openstack-infra | 10:30 | |
| openstackgerrit | Merged openstack-infra/release-tools: stop sending folks to launchpad milestones for release details https://review.openstack.org/266064 | 10:31 |
| *** vivekd has quit IRC | 10:33 | |
| *** vivekd_ has joined #openstack-infra | 10:33 | |
| *** vivekd_ is now known as vivekd | 10:33 | |
| *** dizquierdo has joined #openstack-infra | 10:34 | |
| *** otter768 has quit IRC | 10:34 | |
| openstackgerrit | Thierry Carrez proposed openstack/requirements: Add missing license information https://review.openstack.org/264754 | 10:37 |
| *** rfolco has joined #openstack-infra | 10:40 | |
| *** thorst has joined #openstack-infra | 10:46 | |
| *** _et_ has joined #openstack-infra | 10:47 | |
| *** savihou has joined #openstack-infra | 10:48 | |
| *** jistr has joined #openstack-infra | 10:51 | |
| *** rossella_s has quit IRC | 10:52 | |
| *** rossella_s has joined #openstack-infra | 10:52 | |
| *** thorst has quit IRC | 10:53 | |
| *** markvoelker has joined #openstack-infra | 10:55 | |
| *** markvoelker has quit IRC | 11:00 | |
| *** daemontool_ has joined #openstack-infra | 11:02 | |
| *** sputnik13_ has joined #openstack-infra | 11:02 | |
| *** aysyd has joined #openstack-infra | 11:03 | |
| *** daemontool has quit IRC | 11:05 | |
| *** jaosorior has quit IRC | 11:05 | |
| *** sputnik13 has quit IRC | 11:05 | |
| *** jaosorior has joined #openstack-infra | 11:05 | |
| *** jaosorior has quit IRC | 11:06 | |
| *** jaosorior has joined #openstack-infra | 11:07 | |
| *** lindsayk has quit IRC | 11:08 | |
| *** markvan has quit IRC | 11:09 | |
| *** markvan has joined #openstack-infra | 11:10 | |
| *** hughsaunders has quit IRC | 11:10 | |
| jklare | hi, can anybody help me out with cleaning up the groups/acls and permissions for the openstack-chef project? | 11:12 |
| *** lindsayk has joined #openstack-infra | 11:12 | |
| robcresswell | Hello! So I recently realised that the automatic py27 job that Horizon runs is duplicating a tox task (py27dj18) so is a waste of resources. My initial thought was to remove the dj18 task, but then when we move to django 1.9, we would have to reinstate it | 11:12 |
| *** hughsaunders has joined #openstack-infra | 11:12 | |
| robcresswell | So I am wondering if a better long term solution is to stop using the python-jobs task, and just use tox-{envlist} with a few envs specified for Horizon | 11:13 |
| robcresswell | Is this viable, or have I misunderstood the usage? | 11:13 |
| *** gildub has quit IRC | 11:15 | |
| *** Qiming has joined #openstack-infra | 11:16 | |
| *** lucas-dinner is now known as lucasagomes | 11:18 | |
| *** GheRiver1 has quit IRC | 11:18 | |
| *** aarefiev has quit IRC | 11:20 | |
| *** aarefiev has joined #openstack-infra | 11:21 | |
| *** fhubik is now known as fhubik_brb | 11:21 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project https://review.openstack.org/266843 | 11:21 |
| *** GheRiver1 has joined #openstack-infra | 11:21 | |
| *** GheRivero is now known as GheRivero_ | 11:22 | |
| *** fhubik_brb is now known as fhubik | 11:25 | |
| *** sdake has joined #openstack-infra | 11:29 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project https://review.openstack.org/266843 | 11:33 |
| *** sdake has quit IRC | 11:34 | |
| *** jamespd has quit IRC | 11:34 | |
| *** jamespd has joined #openstack-infra | 11:35 | |
| *** aviau has quit IRC | 11:36 | |
| *** aviau has joined #openstack-infra | 11:38 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/ansible-puppet: Pin to ansible < 2.0, it doesn't support ansible-lint https://review.openstack.org/266857 | 11:41 |
| markus_z | ttx: Hi, do you have a minute for the meeting channel question I asked on the ML? | 11:42 |
| savihou | Hi , any best practices regarding to troubleshooting zuul/gearman configuration? | 11:44 |
| *** fhubik is now known as fhubik_brb | 11:44 | |
| *** kmartin has quit IRC | 11:45 | |
| *** flwang1 has quit IRC | 11:48 | |
| *** flwang1 has joined #openstack-infra | 11:49 | |
| *** thorst has joined #openstack-infra | 11:51 | |
| markus_z | ttx: nevermind, I checked again, and I can chose the same slots on another day of the week without losing any benefit. | 11:51 |
| *** AJaeger has joined #openstack-infra | 11:53 | |
| *** daemontool_ has quit IRC | 11:54 | |
| *** markvoelker has joined #openstack-infra | 11:56 | |
| *** skraynev has quit IRC | 11:57 | |
| *** vgridnev has quit IRC | 11:57 | |
| *** daemontool has joined #openstack-infra | 11:57 | |
| *** esikachev has quit IRC | 11:58 | |
| *** thorst has quit IRC | 11:59 | |
| *** vgridnev has joined #openstack-infra | 11:59 | |
| betherly | hi all! i am in the process of setting up ironic-ui project. so far the governance and new project patches have merged | 12:00 |
| betherly | reading through http://docs.openstack.org/infra/manual/creators.html | 12:00 |
| *** markvoelker has quit IRC | 12:00 | |
| betherly | how do i then go about accessing the repository and being added to groups etc? | 12:01 |
| *** skraynev has joined #openstack-infra | 12:01 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Temporarily pin to ansible==1.9.4 https://review.openstack.org/266864 | 12:01 |
| *** doug-fish has joined #openstack-infra | 12:01 | |
| *** vgridnev has quit IRC | 12:03 | |
| yolanda | betherly, i can help you | 12:04 |
| yolanda | if you send me the link with the project you proposed, i can add you in the groups you need | 12:04 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project https://review.openstack.org/266843 | 12:04 |
| betherly | yolanda: thanks so much! the review links that merged are here https://review.openstack.org/#/c/254845/ and https://review.openstack.org/#/c/254853/ | 12:04 |
| *** esikachev has joined #openstack-infra | 12:04 | |
| *** GheRiver1 has quit IRC | 12:05 | |
| *** salv-orlando has quit IRC | 12:05 | |
| *** baoli has joined #openstack-infra | 12:05 | |
| *** salv-orlando has joined #openstack-infra | 12:05 | |
| *** mase_x200 has joined #openstack-infra | 12:06 | |
| yolanda | betherly, done | 12:06 |
| *** baoli has quit IRC | 12:08 | |
| *** devananda has quit IRC | 12:08 | |
| *** devananda has joined #openstack-infra | 12:13 | |
| *** baoli has joined #openstack-infra | 12:16 | |
| *** baoli has quit IRC | 12:18 | |
| *** vgridnev has joined #openstack-infra | 12:18 | |
| *** rhallisey has joined #openstack-infra | 12:23 | |
| betherly | yolanda: thanks so much :) | 12:25 |
| yolanda | you are welcome | 12:26 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 12:27 |
| *** sdake has joined #openstack-infra | 12:27 | |
| *** sshnaidm has quit IRC | 12:28 | |
| *** vivekd has quit IRC | 12:28 | |
| *** dims has joined #openstack-infra | 12:28 | |
| yolanda | AJaeger, infra-root, can you take a look at https://review.openstack.org/#/c/266864/ ? i want to fix the ansible-lint tests | 12:29 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 12:30 |
| AJaeger | yolanda: It's a big hammer - can | 12:30 |
| AJaeger | yolanda: It's a big hammer - can't we do that only for the lint test? | 12:30 |
| *** gordc has joined #openstack-infra | 12:30 | |
| yolanda | i can create a different builder for ansible-ilnt | 12:30 |
| *** otter768 has joined #openstack-infra | 12:30 | |
| yolanda | ansible-lint | 12:30 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 12:30 |
| robcresswell | Sorry for noise ^^ | 12:31 |
| AJaeger | yolanda: that sounds better - but talk with ansible experts on the proper way here. | 12:31 |
| AJaeger | yolanda: I also prefer with such temporary changes a "# TODO(yolando): Pin to 1.9.4 since ansible-lint does not support 2.0" comment | 12:32 |
| *** vivekd has joined #openstack-infra | 12:32 | |
| yolanda | ok i can amend | 12:32 |
| AJaeger | yes, we have history - but these are things that should be removed later ("temporarily") | 12:32 |
| *** ociuhandu has quit IRC | 12:33 | |
| *** julim has joined #openstack-infra | 12:33 | |
| jklare | AJaeger hi :) since you commented on the patch relating the changes for the openstack-chef groups: do you know what i have to do to make these adaptions to the gerrit groups / create the needed new ones ? | 12:33 |
| ttx | markus_z: good | 12:34 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Temporarily pin to ansible==1.9.4 https://review.openstack.org/266864 | 12:35 |
| *** otter768 has quit IRC | 12:35 | |
| AJaeger | jklare: the new ones will be created as empty once the change merges ;) | 12:35 |
| AJaeger | jklare: I don't know whether we can rename groups, you have to ask others for that. | 12:36 |
| jklare | AJaeger and who will be able to add people to these groups? | 12:36 |
| *** salv-orlando has quit IRC | 12:37 | |
| AJaeger | yolanda: infra admins here will add *you* first and then you can add others | 12:37 |
| *** salv-orlando has joined #openstack-infra | 12:37 | |
| *** fhubik_brb is now known as fhubik | 12:38 | |
| jklare | AJaeger cool, ty | 12:40 |
| AJaeger | I meant jklare instead of yolanda above... | 12:41 |
| AJaeger | yolanda: your change looks fine to me | 12:41 |
| yolanda | AJaeger, i understood it :) | 12:42 |
| *** _amrith_ is now known as amrith | 12:42 | |
| *** kaisers1 has joined #openstack-infra | 12:43 | |
| *** thorst has joined #openstack-infra | 12:43 | |
| AJaeger | robcresswell: you only "declare" the new jobs but you do not use them - don't "call" them. | 12:45 |
| robcresswell | AJaeger: Ah, I see | 12:46 |
| *** kaisers1 has quit IRC | 12:47 | |
| *** hrubi has quit IRC | 12:47 | |
| *** weshay_xchat has joined #openstack-infra | 12:48 | |
| flip214 | mmedvede: ping? | 12:50 |
| *** claudiub has joined #openstack-infra | 12:50 | |
| openstackgerrit | Fabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements https://review.openstack.org/251674 | 12:50 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 12:52 |
| openstackgerrit | nikeshmahalka proposed openstack/requirements: Add google api python client https://review.openstack.org/266893 | 12:52 |
| robcresswell | AJaeger: The aim here is to establish a voting test for dj18, and non-voting for dj19. When we up requirements, I'll remove the dj19 test (as that will be covered by py27) | 12:54 |
| AJaeger | robcresswell: you can also - as you mentioned earlier - remove py27. | 12:55 |
| robcresswell | Yes, I was curious about that but nobody responded (or I missed it) | 12:56 |
| robcresswell | Is it reasonable to remove python-jobs and just use our tox tasks? It would slightly reduce our UT runs, which is always good. | 12:56 |
| *** jcoufal has joined #openstack-infra | 12:57 | |
| *** markvoelker has joined #openstack-infra | 12:57 | |
| *** burgerk has joined #openstack-infra | 12:57 | |
| AJaeger | robcresswell: yes, that's reasonable. | 12:58 |
| AJaeger | And you should first add the job as non-voting to check queue, once it's voting add it to gate as well | 12:58 |
| robcresswell | AJaeger: Great. I'll propose a separate patch for that. | 12:58 |
| *** xyang1 has joined #openstack-infra | 12:58 | |
| robcresswell | AJaeger: Yep, I saw tsufiev comment on it. It's a good point, I'll change now. | 12:58 |
| *** amuller has joined #openstack-infra | 12:59 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 12:59 |
| *** sshnaidm has joined #openstack-infra | 12:59 | |
| *** markvoelker has quit IRC | 13:02 | |
| *** vgridnev has quit IRC | 13:02 | |
| *** links has quit IRC | 13:04 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 13:04 |
| robcresswell | AJaeger: Sorry, that regex was lazy on my part. Not sure what I was thinking. | 13:04 |
| *** ociuhandu has joined #openstack-infra | 13:04 | |
| *** akshai has joined #openstack-infra | 13:04 | |
| AJaeger | robcresswell: please try again - the first regex misses one ) | 13:05 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 13:05 |
| AJaeger | fixed ;) | 13:05 |
| *** vivekd has quit IRC | 13:05 | |
| AJaeger | robcresswell: LGTM, +2 | 13:06 |
| robcresswell | Yeah, my vim autocompletes brackets, sometimes on double )) it gets muddled. | 13:06 |
| robcresswell | AJaeger: Thanks for working with me on that! | 13:06 |
| AJaeger | you're welcome - let's hope Zuul is happy with it as well ;) | 13:06 |
| *** aopincar_ has quit IRC | 13:06 | |
| openstackgerrit | Fabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements https://review.openstack.org/251674 | 13:06 |
| *** akshai has quit IRC | 13:08 | |
| *** rcernin has joined #openstack-infra | 13:09 | |
| openstackgerrit | Fabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements https://review.openstack.org/251674 | 13:10 |
| *** akshai has joined #openstack-infra | 13:11 | |
| *** hrubi has joined #openstack-infra | 13:13 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize entries for neutron subnet https://review.openstack.org/266902 | 13:14 |
| *** sputnik13_ has quit IRC | 13:14 | |
| *** dslevin has quit IRC | 13:15 | |
| *** jsavak has joined #openstack-infra | 13:18 | |
| *** pradk has joined #openstack-infra | 13:19 | |
| AJaeger | robcresswell: gate-django-openstack-auth-tox-py27dj18 is wrong, your project uses _, so use gate-django_openstack_auth-tox-py27djXXX | 13:19 |
| *** trown|outttypeww is now known as trown | 13:22 | |
| robcresswell | AJaeger: ah, dang. Fixing now. | 13:22 |
| *** kgiusti has joined #openstack-infra | 13:22 | |
| *** pradk has quit IRC | 13:23 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth https://review.openstack.org/266878 | 13:23 |
| *** vivekd has joined #openstack-infra | 13:23 | |
| *** baoli has joined #openstack-infra | 13:25 | |
| *** dims has quit IRC | 13:25 | |
| *** dims has joined #openstack-infra | 13:26 | |
| *** dslev has joined #openstack-infra | 13:27 | |
| odyssey4me | do the check and gate pipelines use different nodes from nodepool? we're seeing failures in the gate queue which aren't being exposed in the check queue | 13:28 |
| *** vgridnev has joined #openstack-infra | 13:29 | |
| *** Daisy has joined #openstack-infra | 13:30 | |
| *** markvoelker has joined #openstack-infra | 13:31 | |
| *** ParsectiX has quit IRC | 13:31 | |
| *** amrith is now known as _amrith_ | 13:35 | |
| *** MCoLo has left #openstack-infra | 13:37 | |
| *** baoli_ has joined #openstack-infra | 13:37 | |
| *** edmondsw has joined #openstack-infra | 13:39 | |
| *** berendt has joined #openstack-infra | 13:39 | |
| HeOS | Hello! I have an one question about gerrit and jeepyb, perhaps. Do you have a script to send time-to-time notifications to reviewers which were added into any request? | 13:39 |
| *** burgerk has quit IRC | 13:40 | |
| *** baoli has quit IRC | 13:40 | |
| *** Daisy has quit IRC | 13:41 | |
| *** MCoLo has joined #openstack-infra | 13:41 | |
| *** hdd has joined #openstack-infra | 13:41 | |
| *** Daisy has joined #openstack-infra | 13:42 | |
| *** amitgandhinz has joined #openstack-infra | 13:44 | |
| fungi | odyssey4me: should all be running on the same available image types and providers. have you attempted to correlate the failure to other factors with logstash.openstack.org? | 13:47 |
| *** Daisy has quit IRC | 13:47 | |
| fungi | HeOS: we don't, though also that would get really annoying (i can't even use the default gerrit dashboard because of all the changes people subscribe me to as a reviewer, so the subscription feature is useless for many of us anyway) | 13:47 |
| *** jistr has quit IRC | 13:48 | |
| *** jistr has joined #openstack-infra | 13:49 | |
| *** unicell has joined #openstack-infra | 13:49 | |
| *** unicell1 has quit IRC | 13:50 | |
| yolanda | hi fungi, can you check https://review.openstack.org/266864 ? ansible-lint is broken and tests are failing | 13:50 |
| fungi | mtreinish: the database pruning completed sometime while i was asleep, but rerunning this morning took ~15 minutes (for an idea of runtime when it's more or less keeping up) | 13:51 |
| fungi | yolanda: did we merge something which broke the test? | 13:51 |
| yolanda | fungi, no, but ansible 2.0 landed | 13:51 |
| yolanda | and ansible-lint not compatible | 13:51 |
| fungi | got it. the 2.0 release was recent, i remember that now | 13:52 |
| *** fhubik is now known as fhubik_brb | 13:52 | |
| *** vivekd has quit IRC | 13:53 | |
| fungi | i guess not upgrading until we can run our style checks on it is reasonable, so that we don't introduce stylistic issues we can't test. it's not possible to use 1.9.4 to run ansible-lint on roles which are used by ansible 2.0? | 13:53 |
| *** akshai has quit IRC | 13:54 | |
| fungi | does it also affect our ability to syntax check roles? | 13:54 |
| *** MCoLo has quit IRC | 13:55 | |
| yolanda | fungi, i limited the change to lint only | 13:55 |
| *** dslev has quit IRC | 13:55 | |
| *** hdd has quit IRC | 13:55 | |
| *** mtanino has joined #openstack-infra | 13:55 | |
| yolanda | i tested ansible-lint with ansible 1.9.4 in a role that is used for 2.0, but i'm not sure on the new settings that are brought by ansible 2.0, lint may fail | 13:55 |
| fungi | yep, that works for me (the commit message was just vague and i hadn't pulled up the diff yet) | 13:55 |
| *** ddieterly has joined #openstack-infra | 13:56 | |
| *** eharney has joined #openstack-infra | 13:56 | |
| *** jordanP has joined #openstack-infra | 13:57 | |
| *** Daisy has joined #openstack-infra | 13:57 | |
| *** fhubik_brb is now known as fhubik | 13:58 | |
| *** tlian has joined #openstack-infra | 14:00 | |
| odyssey4me | fungi yolanda FYI https://github.com/willthames/ansible-lint/issues/75 | 14:00 |
| *** akshai has joined #openstack-infra | 14:00 | |
| odyssey4me | fungi I'm busy doing the correlation, but the logstash processing is a little behind - the issue is something new I'm seeing this morning | 14:01 |
| odyssey4me | fungi so far hpcloud and rax-iad | 14:01 |
| fungi | odyssey4me: so also probably something related to 2.0? | 14:02 |
| odyssey4me | fungi http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22One%20or%20more%20undefined%20variables%3A%20'dict%20object'%20has%20no%20attribute%20'ansible_processor_vcpus'%5C%22 | 14:02 |
| *** yamamoto_ has quit IRC | 14:02 | |
| odyssey4me | fungi nope, we used to see this often with ansible 1.6 and below... but haven't seen it since... our tests have ansible capped at < 2.0 | 14:02 |
| odyssey4me | fungi ah, interesting - the last recheck passed on ovh | 14:03 |
| *** bcourt has joined #openstack-infra | 14:03 | |
| odyssey4me | fungi I have a patch going in to work around the issue, I was just wondering whether there was any known underlying hardware differences that could account for the issue | 14:04 |
| *** links has joined #openstack-infra | 14:04 | |
| *** ddieterly has quit IRC | 14:05 | |
| *** dkranz has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily pin to ansible==1.9.4 https://review.openstack.org/266864 | 14:07 |
| *** zz_dimtruck is now known as dimtruck | 14:07 | |
| mordred | fungi: yay! synchronize seems to be fixed for ipv6 in ansible 2 | 14:08 |
| mordred | oh | 14:08 |
| *** links has quit IRC | 14:09 | |
| *** rlandy has joined #openstack-infra | 14:09 | |
| mordred | oh - for project-config we're pinning to 1.9.4 until ansible-lint is fixed | 14:09 |
| mordred | yes | 14:09 |
| mordred | ansible 2.0 on puppetmaster is working nicely | 14:09 |
| yolanda | mordred, it just landed, i saw failures on ansible-lint this morning | 14:09 |
| mordred | yah | 14:09 |
| mordred | ansible-lint definitely no workie with ansible 2.0 | 14:09 |
| *** links has joined #openstack-infra | 14:09 | |
| *** dslev has joined #openstack-infra | 14:10 | |
| pabelanger | yolanda: mordred: still digging into it. Made a little progress yesterday on ansible-lint with 2.0 | 14:10 |
| yolanda | pabelanger, do you know status of ansible-lint fixes? is there any estimate time to support ansible 2.0? | 14:11 |
| *** annegentle has joined #openstack-infra | 14:14 | |
| *** ildikov has quit IRC | 14:15 | |
| *** ccrouch has joined #openstack-infra | 14:16 | |
| pabelanger | yolanda: still a bit away, we've pinned to 1.9 for now | 14:17 |
| yolanda | ok i pinned ansible-lint here to 1.9.4 , that shall work | 14:18 |
| *** salv-orlando has quit IRC | 14:18 | |
| *** salv-orlando has joined #openstack-infra | 14:18 | |
| *** sshnaidm has quit IRC | 14:19 | |
| *** jpeeler has joined #openstack-infra | 14:19 | |
| *** jpeeler has quit IRC | 14:19 | |
| *** jpeeler has joined #openstack-infra | 14:19 | |
| *** GheRivero_ is now known as GheRivero | 14:20 | |
| *** baoli_ has quit IRC | 14:22 | |
| *** baoli has joined #openstack-infra | 14:23 | |
| *** dprince has joined #openstack-infra | 14:24 | |
| *** ayoung has joined #openstack-infra | 14:25 | |
| *** links has quit IRC | 14:25 | |
| *** Daisy has quit IRC | 14:25 | |
| *** links has joined #openstack-infra | 14:25 | |
| *** Daisy has joined #openstack-infra | 14:26 | |
| *** mriedem has joined #openstack-infra | 14:26 | |
| *** peristeri has joined #openstack-infra | 14:27 | |
| *** dimtruck is now known as zz_dimtruck | 14:27 | |
| pabelanger | yolanda: https://github.com/willthames/ansible-lint/pull/79 should have looked there sooner. That's bascially the code I've been working on. So, will attempt to work that PR and see what maintainer wants to do | 14:27 |
| pabelanger | odyssey4me: ^ | 14:27 |
| *** baoli_ has joined #openstack-infra | 14:27 | |
| yolanda | ++ | 14:28 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize vlan name on veth https://review.openstack.org/266942 | 14:28 |
| *** dslev has quit IRC | 14:29 | |
| *** ari has joined #openstack-infra | 14:29 | |
| *** erlon has joined #openstack-infra | 14:29 | |
| *** links has quit IRC | 14:29 | |
| *** baoli has quit IRC | 14:30 | |
| *** links has joined #openstack-infra | 14:30 | |
| *** Daisy has quit IRC | 14:30 | |
| *** otter768 has joined #openstack-infra | 14:31 | |
| *** esker has quit IRC | 14:32 | |
| *** fhubik is now known as fhubik_brb | 14:33 | |
| *** lucasagomes is now known as lucas-hungry | 14:35 | |
| *** otter768 has quit IRC | 14:36 | |
| *** fhubik_brb is now known as fhubik | 14:36 | |
| odyssey4me | pabelanger sorry, I must have missed you in telling everyone about the issue and associated PR :( | 14:37 |
| *** piet has joined #openstack-infra | 14:37 | |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Split changelog on nulls instead of ( https://review.openstack.org/263748 | 14:37 |
| *** dslevin has joined #openstack-infra | 14:38 | |
| pabelanger | odyssey4me: np | 14:38 |
| ari | zaro# Thanks for the review is there someone else that I can get to give me a +2? | 14:39 |
| *** ddieterly has joined #openstack-infra | 14:41 | |
| *** cdent has quit IRC | 14:42 | |
| *** bcourt has quit IRC | 14:44 | |
| *** peristeri has quit IRC | 14:44 | |
| pabelanger | fungi: I know it is likely early, but has anybody here expressed interested in submitting a talk for the upstream developers track in Austin? I seen some -infra specific things already listed on https://etherpad.openstack.org/p/austin-upstream-dev-track-ideas and won't mind offering to aid in presenting | 14:44 |
| *** Swami has joined #openstack-infra | 14:44 | |
| *** ddieterly has quit IRC | 14:46 | |
| fungi | pabelanger: funny you should ask! i was going to put out a specific call for volunteers at the next infra meeting, but you get first pick | 14:47 |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| fungi | early bird eating worms and all that | 14:48 |
| fungi | tasty, tasty conference presentation worms | 14:48 |
| pabelanger | haha | 14:48 |
| pabelanger | fungi: well, I was mostly thinking it would also be a good way to get into the conference talks for OpenStack | 14:49 |
| fungi | you betcha | 14:49 |
| pabelanger | stepping stone if you will | 14:49 |
| mriedem | wtf, nova-net logs aren't even in this failed large ops job http://logs.openstack.org/15/264815/2/check/gate-tempest-dsvm-large-ops/bdb7af7/logs/ | 14:49 |
| *** pradk has joined #openstack-infra | 14:49 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient https://review.openstack.org/266955 | 14:50 |
| pabelanger | fungi: okay, add my name. I would be interested in helping out with that. | 14:50 |
| *** regXboi has joined #openstack-infra | 14:50 | |
| *** levi_b-away has joined #openstack-infra | 14:50 | |
| *** levi_b-away is now known as levi_b | 14:50 | |
| *** levi_b has left #openstack-infra | 14:50 | |
| *** Qiming has quit IRC | 14:51 | |
| fungi | pabelanger: gladly, and thanks for volunteering! | 14:52 |
| *** rossella_s has quit IRC | 14:52 | |
| *** hdd has joined #openstack-infra | 14:52 | |
| *** rossella_s has joined #openstack-infra | 14:52 | |
| *** peristeri has joined #openstack-infra | 14:52 | |
| *** davidlenwell has quit IRC | 14:57 | |
| *** yamamoto has quit IRC | 14:57 | |
| *** mase_x200 has quit IRC | 14:58 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Speed up story GET requests https://review.openstack.org/266961 | 14:58 |
| mmedvede | flip214: ack | 14:58 |
| *** bcourt has joined #openstack-infra | 14:58 | |
| *** davidlenwell has joined #openstack-infra | 14:59 | |
| *** ddieterly has joined #openstack-infra | 14:59 | |
| *** piet has quit IRC | 14:59 | |
| *** GheRiver1 has joined #openstack-infra | 14:59 | |
| *** piet has joined #openstack-infra | 15:00 | |
| flip214 | mmedvede: thanks for replying! | 15:00 |
| flip214 | I submitted a patch for ci-watch about 4 weeks ago, and it got merged. But it's not deployed to tintri.com yet. Is there anything else needed from my side? | 15:01 |
| *** sridhar_ram has quit IRC | 15:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Use scripts from horizon repo in integration job test hooks https://review.openstack.org/263457 | 15:04 |
| mmedvede | flip214: I think apoorva did deploy your fix yesterday, he is the one maintaining ci-watch.tintri.com | 15:04 |
| *** Qiming has joined #openstack-infra | 15:06 | |
| *** links has quit IRC | 15:07 | |
| anteaya | dhellmann: when did you want to discuss release script location? | 15:08 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for devstack setup bug 1457713 https://review.openstack.org/266967 | 15:09 |
| openstack | bug 1457713 in devstack "ERROR: openstack Missing parameter(s)" [Undecided,Confirmed] https://launchpad.net/bugs/1457713 | 15:09 |
| *** _amrith_ is now known as amrith | 15:10 | |
| openstackgerrit | Benjamin Koep proposed openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin https://review.openstack.org/266970 | 15:10 |
| *** burgerk has joined #openstack-infra | 15:10 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [rally] Skip long-running jobs when it possible https://review.openstack.org/265876 | 15:11 |
| *** sshnaidm has joined #openstack-infra | 15:11 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [rally] Skip long-running jobs when it possible https://review.openstack.org/265876 | 15:11 |
| *** HeOS has quit IRC | 15:11 | |
| openstackgerrit | Benjamin Koep proposed openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin https://review.openstack.org/266970 | 15:12 |
| *** fhubik is now known as fhubik_brb | 15:12 | |
| flip214 | mmedvede: thanks, right. the first day I didn't check the JS code, and it's done ;) | 15:12 |
| flip214 | thank you very much! | 15:12 |
| *** yamamoto has joined #openstack-infra | 15:13 | |
| flip214 | if I find any other simple optimizations, is there anything I can do to accelerate deployment? | 15:13 |
| mmedvede | flip214: yw, sorry for the delay. fyi, some of third-party folks also hang out in #openstack-third-party-ci channel | 15:14 |
| *** zz_dimtruck is now known as dimtruck | 15:15 | |
| *** fhubik_brb is now known as fhubik | 15:15 | |
| flip214 | mmedvede: thanks again! | 15:16 |
| mmedvede | flip214: the delay was mainly due to most people having vacations. ci-watch.tintri is not managed by openstack infra team, and is not automatically deployed. Once we have a proper deployment, it should be faster | 15:16 |
| pabelanger | ianw: greghaynes: diskimage-builder installed from RPM (epel7) :) Version 1.4.0, but a start | 15:16 |
| *** fhubik is now known as fhubik_brb | 15:17 | |
| *** levi_b has joined #openstack-infra | 15:17 | |
| *** levi_b has left #openstack-infra | 15:17 | |
| flip214 | mmedvede: ah, okay. I would have thought the people giving +2 (at least +1 on the workflow) would've done the deployment too... | 15:18 |
| *** rbrndt_ has joined #openstack-infra | 15:18 | |
| mmedvede | flip214: if you're interested, I can give you the script to deploy your own ci-watch instance | 15:19 |
| flip214 | mmedvede: thanks, but no thanks ;) | 15:19 |
| mmedvede | haha, ok | 15:20 |
| *** Qiming has quit IRC | 15:20 | |
| *** boris-42 has joined #openstack-infra | 15:20 | |
| *** fhubik_brb is now known as fhubik | 15:20 | |
| *** rbrndt__ has joined #openstack-infra | 15:21 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Speed up story GET requests https://review.openstack.org/266961 | 15:22 |
| *** tdurakov has joined #openstack-infra | 15:22 | |
| tdurakov | hi folks | 15:22 |
| *** rbrndt_ has quit IRC | 15:23 | |
| piet | morning | 15:23 |
| tdurakov | fast question about ansible, have you changed its version recently? | 15:23 |
| odyssey4me | tdurakov ansible 2.0 released yesterday afternoon | 15:24 |
| *** yamamoto has quit IRC | 15:25 | |
| tdurakov | odyssey4me, got strange errors, line: $ANSIBLE subnodes --sudo -f 5 -i "$WORKSPACE/inventory" -m shell -a "mount -t nfs4 -o proto=tcp,port=2049 $primary_node:/ /opt/stack/data/nova/instances/" fails with 2016-01-13 15:04:00.719 | unsupported parameter for module: proto | 15:26 |
| *** zeih has quit IRC | 15:27 | |
| *** rkukura has joined #openstack-infra | 15:27 | |
| *** armax has quit IRC | 15:28 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on refresh token worflow: https://review.openstack.org/266981 | 15:28 |
| *** MarkAtwood has joined #openstack-infra | 15:28 | |
| pabelanger | fungi: clarkb jeblair: nodepool question, do you see any downside to moving reconfigureGearmanClient() to the bottom of the updateConfig()? https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L1892 The reason being, if zuul is down (or not provisioned) gearman blocks, stopping DIBs from being created. | 15:28 |
| pabelanger | going to test it locally now, but figured I ping you all first | 15:28 |
| pabelanger | Hmm, now that I typed that out, I'm unsure if dibs will even build | 15:29 |
| pabelanger | okay, testing | 15:29 |
| ari | zxiiro# Can you give me a review when you have time? - https://review.openstack.org/#/c/263354/ | 15:29 |
| cbader_ | is anyone else see issue with gerrit login. my zuul is getting ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. | 15:30 |
| pabelanger | cbader_: Yup, a result of the gerrit upgrade a few weeks ago | 15:30 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/puppet-log_processor: Allow for python-daemon already defined https://review.openstack.org/266982 | 15:30 |
| *** yamahata has joined #openstack-infra | 15:30 | |
| cbader_ | pabelanger, I have been out the last few weeks, is there a fix I need to put in. | 15:31 |
| igorbelikov | hey folks, can someone review this request to add packetary project https://review.openstack.org/#/c/261540/? I know packetary guys are eager to start using gerrit and this review is hanging without infra comments for a long time | 15:31 |
| clarkb | pabelanger: I dont think thats true before the gearman image builder stuff merges since the devstack tests have no zuul/gearman. Will be rrue after though | 15:31 |
| *** zeih has joined #openstack-infra | 15:31 | |
| pabelanger | clarkb: okay, maybe I am just seeing a long timeout for the socket connection. Either way, testing more locally | 15:32 |
| clarkb | well you tell it not to connect to anything | 15:32 |
| andreaf | yolanda, clarkb: tiny puppet review on log processor when you have a sec: https://review.openstack.org/266982 | 15:32 |
| clarkb | but this all changes today hopefully when I get the image builder stuff in which will need a gearman | 15:33 |
| pabelanger | cbader_: IIRC, it is more a warning then error. nodepool will just log it and work as expected | 15:33 |
| tdurakov | clarkb, hi, do you know what ansible version is now used by ci? | 15:33 |
| cbader_ | clarkb: looks like i have run into a change in gerrit a few weeks ago, I am getting 'ref-replicated' from Gerrit is there a fix you can point me too. | 15:33 |
| AJaeger | tdurakov: unless there are constraints: ansible 2.0 | 15:34 |
| *** tonytan4ever has joined #openstack-infra | 15:34 | |
| pabelanger | clarkb: Ya, in my setup. I want it to connect to gearman, its just that zuul hasn't been provisioned by ansible yet. So, rather then block DIBs for another few minutes, I was looking to reorder the gearmanClient reconfigure | 15:34 |
| AJaeger | tdurakov: ansible-lint uses 1.9.4 - yolanda added a constrainted there | 15:34 |
| pabelanger | clarkb: not a big deal, just have to wait a few minutes | 15:34 |
| clarkb | tdurakov: latest | 15:34 |
| clarkb | cbader_: thats benign, zuul will just ignore those events | 15:35 |
| pabelanger | clarkb: cbader_: maybe we should reduce the log level from ERROR to WARNING | 15:35 |
| *** amitgandhinz has quit IRC | 15:35 | |
| *** FallenPegasus has joined #openstack-infra | 15:35 | |
| clarkb | pabelanger: there is no real point to the reorder after the gearman image builder stuff merges because image building requires gearman at that point | 15:35 |
| tdurakov | AJaeger, clarkb, ok thank you! | 15:35 |
| *** lucas-hungry is now known as lucasagomes | 15:36 | |
| pabelanger | clarkb: Ya, that's what I wanted to confirm | 15:36 |
| *** amitgandhinz has joined #openstack-infra | 15:36 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for devstack setup bug 1457713 https://review.openstack.org/266967 | 15:37 |
| openstack | bug 1457713 in devstack "ERROR: openstack Missing parameter(s)" [Undecided,Confirmed] https://launchpad.net/bugs/1457713 | 15:37 |
| openstackgerrit | Christian Berendt proposed openstack-infra/irc-meetings: Adjust meeting time for US/EU doc-install-team-meeting https://review.openstack.org/266987 | 15:38 |
| clarkb | pabelanger: cbader_ are those unknown event things logged as errors? | 15:38 |
| *** fhubik is now known as fhubik_brb | 15:39 | |
| *** fhubik_brb is now known as fhubik | 15:39 | |
| pabelanger | clarkb: ya | 15:40 |
| *** Sukhdev has joined #openstack-infra | 15:40 | |
| pabelanger | https://github.com/openstack-infra/zuul/blob/master/zuul/connection/gerrit.py#L96 | 15:40 |
| pabelanger | err | 15:40 |
| asselin | Thelo, this is your issue: slave nodes can't access your CI node: Could not resolve host: hps10.ccma.itri.org.tw | 15:40 |
| *** fawadkhaliq has joined #openstack-infra | 15:40 | |
| cbader_ | clarkb, pabelanger, yes as error. 2016-01-13 07:40:27,598 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information. | 15:41 |
| asselin | Thelo, so it will default to review.openstack.org. If you look at the script that checks out, there's a checkout order. | 15:41 |
| pabelanger | cbader_: thanks, save me from digging up my error logs | 15:41 |
| *** nmagnezi has quit IRC | 15:41 | |
| asselin | cbader_, I saw that issue yesterday in my test ci. When upgrading to the latest zuul they went away. | 15:41 |
| pabelanger | clarkb: cbader_ I can get a patch up to reduce it to a warning | 15:42 |
| asselin | cbader_, or maybe that's coincidence.... | 15:42 |
| cbader_ | pabelanger, I made a change in my nodepool.nodepool.yaml and did an ./install_manager.sh which I guess updated my zuul and nodepool and now can't get nodepool vm's to start | 15:43 |
| pabelanger | cbader_: if you run nodepoold -d manually, what do you see? | 15:44 |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| *** gongysh_ has joined #openstack-infra | 15:45 | |
| cbader_ | pabelanger: I get IOError: [Errno 13] Permission denied: '/etc/nodepool/secure.conf' | 15:46 |
| cbader_ | pabelanger: and locks up | 15:46 |
| waynr | electrofelix: so here's a shorter compare xml time: https://review.openstack.org/#/c/262668/ | 15:46 |
| *** berendt has quit IRC | 15:46 | |
| pabelanger | cbader_: so, need to fix your permission error | 15:46 |
| cbader_ | pabelanger: don't know what would have changed to cause this error. | 15:47 |
| Thelo | asselin: thanks I found the issue earlier this evening, however I dont quite get why we get the code from the CI server and the role of zuul-merger | 15:47 |
| *** armax has joined #openstack-infra | 15:48 | |
| pabelanger | cbader_: not sure how you install nodepool. Are you using the same user as the service would? | 15:48 |
| pabelanger | cbader_: either way, looks to be a simple fix | 15:48 |
| tdurakov | clarkb, it looks like there is regression with new ansible | 15:48 |
| openstackgerrit | Merged openstack/requirements: Change OSprofiler constraints to >=0.4.0 https://review.openstack.org/266264 | 15:49 |
| asselin | Thelo, so the ci server, in particular the zuul merger, is responsible to prepare any git references needed by the project. | 15:50 |
| asselin | Thelo, so if you make a change to the project 'cinder', it is the zuul-merger that will have that change prepared locally so that your ci jenkin slave can test the right patch. | 15:51 |
| waynr | i've seen compare xml time on master branch take as long as 9m | 15:51 |
| cbader_ | does anyone know where the CI Watch get the url for the logs it attaches too? I had to change my log file server and the CI Watch is not connecting to the correct URL? | 15:51 |
| *** MarkAtwood has quit IRC | 15:51 | |
| asselin | Thelo, furthermore, if your cinder patch has a 'depends-on' to a nova patch, the zuul merger will prepare 2 refs, one for cinder and one for nova. | 15:51 |
| asselin | Thelo, in this way, you ci- jenkins slave can get both patches and run the correct test. | 15:51 |
| *** jpr has joined #openstack-infra | 15:52 | |
| asselin | cbader_, it gets if from the gerrit comment | 15:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Log 'Received unrecognized event type' as warning https://review.openstack.org/266998 | 15:53 |
| *** jpr has left #openstack-infra | 15:53 | |
| Thelo | asselin: Got it, thanks | 15:53 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Add neutron -plus job template for advanced features https://review.openstack.org/247697 | 15:53 |
| *** mgould has joined #openstack-infra | 15:53 | |
| *** jpr has joined #openstack-infra | 15:53 | |
| cbader_ | asselin: sorry don't know where the gerrit comment is configured. | 15:54 |
| *** matrohon has quit IRC | 15:54 | |
| asselin | cbader_, I believe it is from zuul: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.conf.erb#n20 | 15:55 |
| asselin | cbader_, /etc/zuul/zuul.conf 'url_pattern' | 15:56 |
| *** bcourt has quit IRC | 15:56 | |
| *** sridhar_ram has joined #openstack-infra | 15:56 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Add neutron -plus job template for advanced features https://review.openstack.org/247697 | 15:56 |
| asselin | cbader_, http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n215 | 15:57 |
| asselin | cbader_, if that value is correct, you probably need to restart zuul | 15:58 |
| *** bardia has joined #openstack-infra | 15:58 | |
| cbader_ | asselin: the ip in the /etc/zuul/zuul.conf is correct. | 15:58 |
| *** yamamoto has quit IRC | 15:59 | |
| *** devananda has quit IRC | 15:59 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Improve "add card" UI https://review.openstack.org/267002 | 16:00 |
| Zara | whoa I can see the speedup | 16:00 |
| Zara | even on my tiny instance | 16:00 |
| *** kmartin has joined #openstack-infra | 16:00 | |
| *** sputnik13 has joined #openstack-infra | 16:01 | |
| anteaya | Zara: nice | 16:01 |
| Zara | hah, oops, I meant to type that in #storyboard but it still applies! xD | 16:02 |
| *** devananda has joined #openstack-infra | 16:02 | |
| clarkb | tdurakov: do you have more details than that? | 16:02 |
| tdurakov | yep | 16:02 |
| tdurakov | i used adhoc shell module | 16:03 |
| tdurakov | when there is '=' in arguments it's treated as key-value | 16:03 |
| *** jsavak has quit IRC | 16:03 | |
| *** Jeffrey4l has joined #openstack-infra | 16:04 | |
| tdurakov | clarkb, $ANSIBLE subnodes --sudo -f 5 -i "$WORKSPACE/inventory" -m shell -a "mount -t nfs4 -o proto\=tcp,port\=2049 $primary_node:/ /opt/stack/data/nova/instances/" | 16:04 |
| tdurakov | need to be escaped | 16:04 |
| *** jsavak has joined #openstack-infra | 16:04 | |
| clarkb | tdurakov: it is possible that was intentional, they did a 2.0 release because not everything is backard compatoble | 16:05 |
| clarkb | but you best bet is to ask asnible | 16:05 |
| cbader_ | asselin: I am getting the can not get account information still after restarting zuul and zuul-merger is there a fix for that? | 16:05 |
| tdurakov | clarkb, will try to ask | 16:05 |
| tdurakov | it affects my job that I'm working on in experimental pipeline, so not urgent at this moment, but it could be in other jobs/hooks^( | 16:06 |
| asselin | cbader_, not sure where you see that.... | 16:07 |
| dhellmann | anteaya, fungi, jeblair: do you have some time to talk about those release scripts quickly? I think ttx have come up with a plan that reduces my concern about moving some of them into project-config | 16:07 |
| dhellmann | * ttx and i | 16:07 |
| *** austin81 has joined #openstack-infra | 16:07 | |
| anteaya | dhellmann: /me is here and standing by | 16:07 |
| cbader_ | asselin: in the /var/log/zuul/zuul.log | 16:07 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 16:07 | |
| jeblair | dhellmann: o/ | 16:08 |
| cbader_ | asselin: getting this, 2016-01-13 08:08:03,648 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit. Can not get account information. | 16:08 |
| dhellmann | anteaya , jeblair : let me run this past you, and I can catch up with fungi later if I need to | 16:08 |
| asselin | cbader_, not sure if that's impacting any real functionality | 16:08 |
| anteaya | dhellmann: /me listens | 16:09 |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on refresh token worflow: https://review.openstack.org/266981 | 16:09 |
| asselin | cbader_, seems more of a warning if anything | 16:09 |
| dhellmann | so my issues with moving all of the release tools into project-config are that (a) some are complex and change frequently and (b) converting them to work to use system packages for things that aren't packaged in ubuntu yet would be a hassle | 16:09 |
| dhellmann | we think, now, that with some changes to just the script that submits the tags, we can move that one into project-config | 16:10 |
| *** salv-orl_ has joined #openstack-infra | 16:10 | |
| dhellmann | all of the other stuff related to generating release announcements contains the most complexity, and that we should be able to run on a non-privileged node so we shouldn't need to put it into project config or worry about using a virtualenv | 16:10 |
| fungi | dhellmann: i'm around now, sorry | 16:11 |
| fungi | catching up | 16:11 |
| dhellmann | which leaves the script that adds comments related to released bugs, which falls somewhere in between: it's complex and needs launchpad tools and credentials | 16:11 |
| dhellmann | it's likely all of its dependencies are available as debs, though | 16:11 |
| dhellmann | fungi : np | 16:11 |
| dhellmann | so the idea is to do the refactoring of the tools that generate the tags and move those to project-config | 16:12 |
| dhellmann | and then see about the release announcement stuff as a separate unprivileged job | 16:12 |
| dhellmann | though, do the unprivileged nodes have exim? | 16:12 |
| *** bcourt has joined #openstack-infra | 16:12 | |
| dhellmann | or some other way to send email, I suppose, it doesn't have to be exim | 16:13 |
| *** ldnunes has joined #openstack-infra | 16:13 | |
| jeblair | dhellmann: yes they do | 16:13 |
| *** salv-orlando has quit IRC | 16:13 | |
| dhellmann | ok, cool | 16:13 |
| fungi | yeah, having them send e-mail to the listserv is probably fine | 16:13 |
| dhellmann | yeah, we might have some of the oslo releases announced to python-announce, too, but we don't do that right now | 16:13 |
| fungi | it's mostly things with access to teh signing key and lp account credentials we need to audit more closely | 16:14 |
| jeblair | dhellmann: the trick is that if you submit a message locally, the node needs to live long enough to deliver it | 16:14 |
| dhellmann | jeblair : I could have the script watch the exim log, I guess? feels dirty, though. | 16:14 |
| fungi | it might make more sense for the job to smtp directly to lists.o.o and block the job completion until it gets accepted? | 16:14 |
| dhellmann | yeah, that works, too | 16:15 |
| dhellmann | ok, I'll make a note and make that change | 16:15 |
| jeblair | dhellmann: i would not expect delivery to anywhere but lists.o.o to work for now (and even then, we may have to tweak something); delivery to other sites will likely require a smarthost, and, heh, probably running on a trusted slave with smarthost credentials. | 16:15 |
| *** katyafervent is now known as katyafervent_awa | 16:15 | |
| *** katyafervent_awa is now known as katyafervent | 16:15 | |
| fungi | since it's not delivering to multiple destinations, a local mta would mostly be overkill for this | 16:15 |
| dhellmann | ok, we can worry about other announcements later, then | 16:16 |
| jeblair | i think thats best | 16:16 |
| *** rhallisey has quit IRC | 16:16 | |
| *** trown is now known as trown|brb | 16:17 | |
| dhellmann | so that leaves figuring out what to do with the launchpad script. it runs out of the release tagging script right now, and we could keep doing that if I can fix it up to not need a virtualenv | 16:17 |
| dhellmann | it would be good to do that only after the tarball is built, though, and that either means a separate job or having the tagging job wait for the packaging job | 16:17 |
| jeblair | dhellmann: i think i did overstate the delay problem with slave_scripts yesterday... while it's true that if they are used on throwaway nodes it can take days for an image update to complete and for them to show up there -- when they are used on a static slave, that's usually not a problem; puppet will keep them updated so it should be <30 mins after merge | 16:18 |
| *** trown|brb is now known as trown | 16:18 | |
| dhellmann | ok, that's good to know | 16:18 |
| *** dims_ has joined #openstack-infra | 16:18 | |
| anteaya | I also didn't know that, thanks jeblair | 16:18 |
| *** yamamoto has joined #openstack-infra | 16:18 | |
| dhellmann | we pretty regularly discover some odd behavior because all of these repos are slightly different, and have to make adjustments for it | 16:19 |
| fungi | yeah, i mentioned that yesterday, but it eventually becomes a general problem in zuulv3 | 16:19 |
| dhellmann | that's *mostly* in the release notes, though | 16:19 |
| *** rhallisey has joined #openstack-infra | 16:19 | |
| jeblair | fungi: yep | 16:19 |
| anteaya | fungi: sorry I missed that | 16:19 |
| mtreinish | fungi: ok, cool yeah I noticed it was finished pruning at somepoint before I went to sleep last night | 16:19 |
| dhellmann | and if we can send release nodes from a job where the code is pulled from the release-tools repo as it's needed, then we have no delay | 16:19 |
| *** rajinir has joined #openstack-infra | 16:19 | |
| mtreinish | I'll drop my WIP from the patch adding the cron job to run it then | 16:19 |
| mgould | hi everyone! are there any requirements cores about who could take a look at https://review.openstack.org/#/c/266320/ ? | 16:19 |
| mgould | one-line patch to bump minor python-ironicclient version | 16:20 |
| *** amitgandhinz has quit IRC | 16:20 | |
| dhellmann | fungi, jeblair: the release notes script needs a repo and version as input, and then it scans the git tree for the history (among other things). how ugly is it to have a job triggered when a patch lands in the releases repository wait for the tarball to be built before generating the release notes? | 16:21 |
| *** dims has quit IRC | 16:21 | |
| *** flepied has quit IRC | 16:21 | |
| dhellmann | I think we already have a script for waiting for a tarball, so I'm really asking if there's an objection to doing that in CI vs. locally like we've been doing it | 16:21 |
| *** flepied has joined #openstack-infra | 16:21 | |
| fungi | dhellmann: we could make it depend on the tarball job the same way the upload job does | 16:21 |
| sfinucan | anteaya: fungi: Can we pick up the discussion from yesterday on this patch series? https://review.openstack.org/#/c/258984/15 | 16:22 |
| jeblair | dhellmann: a job should run after a patch landes in the release repo but it should wait until a tarball job from say the nova repo completes before running? | 16:22 |
| dhellmann | fungi : how would that job know what version was being built? | 16:22 |
| fungi | right now for things released to pypi we have one job generate the tarball/wheel, then a second job on another worker upload them to pypi if the first job succeeds | 16:22 |
| anteaya | sfinucan: perhaps after we finish with dhellmann's thing | 16:22 |
| *** ihrachys has quit IRC | 16:22 | |
| dhellmann | jeblair : right | 16:22 |
| *** sdake_ has joined #openstack-infra | 16:22 | |
| fungi | oh, patch to the release repo but tarball for another repo | 16:22 |
| sfinucan | anteaya: Oh, of course :) | 16:22 |
| dhellmann | right | 16:22 |
| clarkb | jeblair: our first non gate dependent pipeline? | 16:23 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Add cron job to run weekly prune the data set of db https://review.openstack.org/176564 | 16:23 |
| SergeyLukjanov | fungi, jeblair, hi, could you please help. we've found out in sahara team that we created a new repo (sahara-scenario) incorrectly and we need to have repo named sahara-tests with git history (currently created empty repo), do you think it'll be okay to request new repo and move old one to attic eventually? other option is to rename repo and re-push with --force? what do you think guys about it? | 16:23 |
| fungi | that's a little weirder to work out | 16:23 |
| *** angdraug has joined #openstack-infra | 16:23 | |
| jeblair | fungi, dhellmann: neat. i can't think of any way to do that right now, but cross-repo job dependencies may be a fun thing to explore. :) | 16:23 |
| anteaya | sfinucan: thank you :) | 16:23 |
| jeblair | clarkb: do you think a dependent pipeline could do it...? hrm.... | 16:23 |
| clarkb | oh except one job has to complete before the other whereas dependent pipeline would parallel | 16:23 |
| clarkb | hrm | 16:23 |
| dhellmann | jeblair, fungi : ok, I can do it now with a thing that polls until the tarball exists on the server, so if that's ok as an interim solution I could help explore "better" solutions down the road, too | 16:23 |
| jeblair | dhellmann: what causes the tarball job to run? tagging the repo? | 16:24 |
| fungi | SergeyLukjanov: can we fix the git history on sahara-scenario now and rename sahara-scenario to sahara-tests during a maintenance (we already have another rename requested we need to schedule anyway) | 16:24 |
| dhellmann | jeblair, fungi: my not quite secret goal is to have all releases going through this machinery by early in the next dev cycle, so I'd like to identify short-term and long-term solutions that support that | 16:24 |
| dhellmann | jeblair : yeah | 16:24 |
| *** tonytan4ever has quit IRC | 16:24 | |
| jeblair | dhellmann: and that's done by the previously mentioned tagging script ,right? | 16:24 |
| dhellmann | jeblair : right | 16:24 |
| jeblair | dhellmann: so -- commit to releases; tag pushed; tarball built; release notes job run ? | 16:25 |
| SergeyLukjanov | fungi, I think yes - actually we have an empty repo now and need to init it from the external repo | 16:25 |
| dhellmann | jeblair : right | 16:25 |
| SergeyLukjanov | fungi, do you think it'll be okay? | 16:25 |
| *** sdake has quit IRC | 16:25 | |
| *** sdake_ is now known as sdake | 16:25 | |
| jeblair | dhellmann, fungi: can the release notes job be attached to the _project_ rather than release repo? | 16:26 |
| fungi | dhellmann: ahh, so change merges to release repo, release repo job generates and pushes a tag for nova into gerrit, tag on nova triggers nova tarball job... and we want another release repo job to run after that completes | 16:26 |
| clarkb | jeblair: if we made a dependent pipeline with a window of 1 that never shifted that may work | 16:26 |
| clarkb | jeblair: major hacks though | 16:26 |
| dhellmann | jeblair : maybe? it needs the repo and a tag as input, so if that's available there may be a way? | 16:27 |
| dhellmann | I suppose we could use git describe to determine the tag | 16:27 |
| fungi | SergeyLukjanov: yeah, so push --force the correct content into the empty repo for now, then add it to the renames list on the meeting agenda to be scheduled. we can pick a date for the next renames batch when we meet on this coming tuesday | 16:27 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize vlan name on veth https://review.openstack.org/266942 | 16:27 |
| dhellmann | as long as the local copy of the working repo is checked out to the tagged commit | 16:27 |
| jeblair | dhellmann: it would have the tag (and commit that was tagged) in nova as well as the project, 'nova', as input... so it could go from there i think | 16:27 |
| dhellmann | ok, I think we can make that part work, then | 16:28 |
| jeblair | dhellmann: no need for git describe, it will all be in ZUUL_ vars | 16:28 |
| SergeyLukjanov | fungi, cool, thx, I'll add it to the agenda | 16:28 |
| fungi | yeah, basically it would be a nova job which depends on the nova tarball job and fires the necessary machinery (the tag name is in the zuul parameters already) | 16:28 |
| dhellmann | jeblair : good, that seems more accurate | 16:28 |
| jeblair | dhellmann, fungi, clarkb: that means the release notes repo can just wait on the nova tarball release job then | 16:28 |
| *** ildikov has joined #openstack-infra | 16:28 | |
| jeblair | gat | 16:29 |
| jeblair | dhellmann, fungi, clarkb: that means the release notes job can just wait on the nova tarball release job then | 16:29 |
| fungi | yeah, parse error there | 16:29 |
| jeblair | i have not had breakfast yet | 16:29 |
| fungi | np, that makes much more sense | 16:29 |
| dhellmann | yeah, that makes more sense, ok cool | 16:29 |
| *** aopincar_ has joined #openstack-infra | 16:30 | |
| *** yamamoto has quit IRC | 16:30 | |
| *** yaume_ has quit IRC | 16:30 | |
| dhellmann | fungi, jeblair : ok, the final question was how do we know when these jobs pass/fail? is there some way to see them easily in a dashboard or something? | 16:30 |
| *** yaume has joined #openstack-infra | 16:30 | |
| *** danpb has joined #openstack-infra | 16:30 | |
| anteaya | their logs would be in post logs wouldn't they? | 16:31 |
| fungi | dhellmann: not yet, other than looking at their logs or graphite or catching them in the zuul status, but there are efforts underway to add a better dashboard to things like post/release/periodic job logs | 16:31 |
| fungi | so that they're a little more discoverable | 16:31 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: set keystone's legacy_drivers job to voting https://review.openstack.org/266582 | 16:31 |
| anteaya | oh where is that dashboard going? | 16:31 |
| fungi | right now they're just going to be in the usual log path for teh repo+pipeline+jobname | 16:32 |
| *** otter768 has joined #openstack-infra | 16:32 | |
| *** abregman has quit IRC | 16:32 | |
| dhellmann | fungi : ok, when the time comes I'll get those path details from you. The only pattern I know is based on SHA values. | 16:33 |
| fungi | anteaya: i think the hope was to get it integrated into the health dashboard, though rbradfor was also working on a proof-of-concept to do something similar for coverage jobs specifically | 16:33 |
| anteaya | fungi: okey dokey, thanks | 16:33 |
| dhellmann | fungi, jeblair: I think you've answered my concerns. thanks for your time! | 16:34 |
| fungi | dhellmann: thanks for making time to continue the discussion! | 16:34 |
| anteaya | dhellmann: thank you | 16:34 |
| anteaya | fungi clarkb jeblair danpb would like to be added to a new gerrit group | 16:35 |
| anteaya | os-vif-core | 16:35 |
| anteaya | if someone could assist | 16:35 |
| anteaya | sfinucan: you are up | 16:35 |
| *** mpavone has quit IRC | 16:35 | |
| sfinucan | anteaya: yay? | 16:35 |
| anteaya | https://review.openstack.org/#/admin/groups/1175,members link to os-vif-core | 16:36 |
| sfinucan | :) | 16:36 |
| anteaya | sfinucan: thanks for you patience | 16:36 |
| anteaya | sfinucan: lets start with a link to the parent patch of that series from yesterday and see where we are | 16:36 |
| sfinucan | anteaya: thanks for helping with this | 16:36 |
| sfinucan | sure | 16:36 |
| sfinucan | anteaya: this is it https://review.openstack.org/#/c/257956/15 | 16:36 |
| anteaya | sfinucan: I do my best to listen to whatever I can | 16:37 |
| sfinucan | it's broken (see comment), but that's not the issue | 16:37 |
| *** otter768 has quit IRC | 16:37 | |
| rbradfor | anteaya, I have a proof of concept for coverage jobs that builds a page to the logs from the zuul status (as the index page did not provide a link to the correct logs) | 16:37 |
| *** Guest22649 has quit IRC | 16:38 | |
| anteaya | sfinucan: so the first two patches in the series fail unit tests and docs | 16:38 |
| *** Guest91320 is now known as Vivek | 16:38 | |
| anteaya | sfinucan: then the third patch https://review.openstack.org/#/c/257995/16 gives you this patch depends on a patch that fails to merge | 16:39 |
| *** Vivek is now known as Guest14289 | 16:39 | |
| fungi | rbradfor: hopefully that will be fixed once https://review.openstack.org/#/q/status:open+topic:enable_swift are merged | 16:39 |
| sfinucan | anteaya: Yes, and the second patch is new to this "revision", i.e. patch 3 used to be patch 2 | 16:39 |
| jeblair | danpb: who are you in gerrit? | 16:39 |
| anteaya | sfinucan: now I can't say definitively but I personally would be curious if your fails to merge error goes away if the parent patch could pass unit and doc tests | 16:39 |
| danpb | jeblair: email addr berrange@redhat.com | 16:40 |
| *** Guest14289 is now known as VivekVC | 16:40 | |
| *** Hal has joined #openstack-infra | 16:40 | |
| jeblair | danpb: done! | 16:40 |
| anteaya | jeblair: danpb thank you | 16:40 |
| *** Hal is now known as Guest70094 | 16:40 | |
| rbradfor | fungi, woot! | 16:40 |
| *** bhunter71 has joined #openstack-infra | 16:40 | |
| danpb | jeblair: anteaya great, thank you both for your help | 16:40 |
| anteaya | danpb: thanks for your patience | 16:40 |
| sfinucan | anteaya: So patchset 6 of 257995 depended on patchset 6 of 257956 | 16:41 |
| anteaya | sfinucan: okay so I don't have access to the behind the scenes logs for gerrit and zuul | 16:41 |
| sfinucan | and the latter passed Jenkins tests | 16:41 |
| anteaya | sfinucan: but I'm going to recommend getting unit and doc tests passing on the parent patch | 16:41 |
| sfinucan | but the former still gave that failed to merge issue | 16:41 |
| *** bpokorny has joined #openstack-infra | 16:42 | |
| *** Guest70094 has quit IRC | 16:42 | |
| sfinucan | anteaya: Sure, I can do that. I don't think it's the issues, however :/ | 16:42 |
| anteaya | sfinucan: well at the very least it will make the issue clearer if that isn't it | 16:43 |
| sfinucan | anteaya: True. OK, I'll do that now and follow up tomorrow :) | 16:43 |
| fungi | anybody remember what happened to this change? https://review.openstack.org/#/c/175880/https://review.openstack.org/175880 the api claims it's in a merged state but it looks incomplete (no successfully merged comment, no state displayed in the webui) | 16:43 |
| fungi | grr, https://review.openstack.org/175880 | 16:43 |
| anteaya | sfinucan: thank you, and thanks for asking when we had time to spend with you, makes me so much calmer :) I'm grateful | 16:44 |
| fungi | i need to fix my middle-click to stop bouncing | 16:44 |
| anteaya | fungi: I am not familiar with that change at all | 16:44 |
| jeblair | fungi: no idea | 16:44 |
| *** cnesa has quit IRC | 16:44 | |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| jeblair | fungi: when was the rolled back gerrit upgrade? | 16:45 |
| fungi | jeblair: may 11 i thought | 16:45 |
| anteaya | may I think | 16:45 |
| fungi | so this would have been a couple weeks prior | 16:45 |
| fungi | i don't see any mention of it in logs of this channel (until just now) | 16:46 |
| anteaya | yeah may 9th was the upgrade | 16:46 |
| *** samuelBartel has quit IRC | 16:47 | |
| anteaya | there is the rollback email: http://lists.openstack.org/pipermail/openstack-dev/2015-May/063877.html | 16:47 |
| openstackgerrit | Guido Günther proposed openstack/python-jenkins: Add support for handling promotions https://review.openstack.org/258032 | 16:47 |
| jeblair | fungi, anteaya: considering there is a verified+2 on apr26, it _probably_ merged on that date | 16:48 |
| fungi | anyway, it's past my caring about threshold since it's just turning up in my tests to validate the new atc listing script against the september elections. it's not in scope for passes to the next summit and won't be in scope for the next election either | 16:48 |
| anteaya | jeblair: I support that logic | 16:48 |
| fungi | also it's from someone who was a later contributor to the same repo, so doesn't skew my results | 16:48 |
| jeblair | fungi, anteaya: if we care to examine further, we may be able to pull data from backups. | 16:48 |
| fungi | just an anomaly i was running down from my warning output | 16:49 |
| anteaya | I'm satisfied | 16:49 |
| *** arxcruz has quit IRC | 16:49 | |
| jeblair | btw, our backup partition is full so we're probably not currently backing anything up. | 16:49 |
| electrofelix | waynr: so looks like it's just some variation due to build slave :) | 16:49 |
| jeblair | no one ever did step up to work on that :( | 16:49 |
| anteaya | if this was a one time patch I'd want to follow up for elections at least, but jd is a frequent contributor | 16:49 |
| fungi | yargh | 16:49 |
| *** bradjones has quit IRC | 16:49 | |
| anteaya | jeblair: oh :( | 16:50 |
| anteaya | where is the backup partition on cacti, I must have missed that | 16:50 |
| *** bradjones has joined #openstack-infra | 16:50 | |
| *** bradjones has quit IRC | 16:50 | |
| *** bradjones has joined #openstack-infra | 16:50 | |
| *** vgridnev has quit IRC | 16:50 | |
| fungi | jeblair: what did we need there? just more space? pruning retention (is that possible with bup)? something else? | 16:50 |
| jeblair | i'm sorry if this rubs anyone the wrong way, but we really need people to care about boring important stuff. | 16:51 |
| anteaya | it doesn't rub me the wrong way | 16:51 |
| anteaya | and i agree with you | 16:51 |
| *** fhubik has quit IRC | 16:51 | |
| *** rcernin has quit IRC | 16:51 | |
| anteaya | and I'm glad you stated it as such | 16:51 |
| jeblair | fungi: i'm not sure if we can prune retention :) | 16:52 |
| *** tiswanso has joined #openstack-infra | 16:52 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show Project Groups on search page https://review.openstack.org/267027 | 16:52 |
| fungi | right, bup storing it as git history | 16:52 |
| fungi | we do have people who care about boring important stuff, but i agree we have more boring important stuff than we have people caring about it | 16:52 |
| pleia2 | good morning | 16:53 |
| jeblair | fungi: yes, and, when faced point blank with the question of "will anyone take the time to validate our backup regime?" the answer has been silence. | 16:53 |
| * jd__ is glad he will still have his ATC pass ;) | 16:53 | |
| *** amuller is now known as amuller_afk | 16:54 | |
| *** danpb has left #openstack-infra | 16:54 | |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| anteaya | pleia2: good morning | 16:54 |
| *** mtanino has quit IRC | 16:54 | |
| anteaya | jd__: yes thank you for being today's anomoly in fungi's script | 16:54 |
| *** harlowja_at_home has joined #openstack-infra | 16:54 | |
| fungi | jeblair: we did have a volunteer, but the volunteer sort of disappeared after the validation plan was half written and i didn't follow up on it | 16:55 |
| jeblair | fungi: i will see about adding more space | 16:55 |
| jeblair | fungi: i will also attempt some restores to see if it's broken or not | 16:55 |
| jeblair | fungi: confirmed -- prune is not a feature of bup | 16:56 |
| *** vgridnev has joined #openstack-infra | 16:56 | |
| *** vgridnev has quit IRC | 16:56 | |
| fungi | jeblair: thanks, just managed to get logged in myself (my ssh key on there is still old | 16:57 |
| Clint | "yet" | 16:57 |
| *** ifarkas has quit IRC | 16:57 | |
| fungi | in good news, /opt/backups is already a logical volume | 16:59 |
| *** MarkAtwood has joined #openstack-infra | 16:59 | |
| jeblair | yeah, i'm cinder expanding it now | 17:00 |
| rbradfor | I'm seeing some infra help in a merged patch #264977 that updated docs and includes code examples (http://docs.openstack.org/developer/oslo.log/usage.html). I'm getting a forbidden page e.g. http://docs.openstack.org/developer/oslo.log/_downloads/python_logging.py | 17:00 |
| fungi | so as long as we don't have corruption, we should hopefully just be able to add a second cinder volume behing it | 17:00 |
| *** HeOS has joined #openstack-infra | 17:00 | |
| fungi | er, behind | 17:00 |
| anteaya | yay good news | 17:00 |
| cbader_ | I am trying to determine if my CI is functional and I tried to kickoff a previous check with the zuul command but I am not seeing the patch start or any errors, anywhere else I can look? | 17:00 |
| *** jlanoux has quit IRC | 17:00 | |
| fungi | jeblair: thanks for shifting gears to work on that | 17:00 |
| jeblair | np | 17:01 |
| anteaya | rbradfor: I confirm I am also seeing forbidden at that link | 17:01 |
| cbader_ | I ran the command: sudo zuul enqueue --trigger gerrit --pipeline check --project openstack/cinder --change 221441,14 it worked before has something changed so this doesn't work anymore. | 17:01 |
| *** savihou has quit IRC | 17:02 | |
| rbradfor | anteaya, what's interesting, is the docs embed files, and that is visible (i.e the first code example), so the file I assume is actually there. | 17:02 |
| *** salv-orl_ has quit IRC | 17:03 | |
| cbader_ | If I Cntl C the line I see this File "/usr/local/lib/python2.7/dist-packages/zuul/rpcclient.py", line 57, in enqueue | 17:03 |
| cbader_ | return not self.submitJob('zuul:enqueue', data).failure | 17:03 |
| cbader_ | File "/usr/local/lib/python2.7/dist-packages/zuul/rpcclient.py", line 45, in submitJob | 17:03 |
| cbader_ | time.sleep(0.1) | 17:03 |
| rbradfor | anteaya, or perhaps it's generated and not using the actual example file. | 17:03 |
| *** salv-orlando has joined #openstack-infra | 17:03 | |
| waynr | electrofelix zxiiro zaro clarkb last night I gave a presentation at a local meetup on the JJB 2.x API and a tool demonstrating its use, if you're interested the presentation and an example Python-defined jenkins pipeline can be found here: https://github.com/waynr/jjb-2.x-tutorial | 17:04 |
| electrofelix | waynr: thanks, taking a look | 17:04 |
| anteaya | cbader_: please use a paste | 17:04 |
| zxiiro | waynr: cool thanks for sharing | 17:04 |
| anteaya | rbradfor: both good questions | 17:04 |
| * zxiiro adds it to his reading list | 17:05 | |
| cbader_ | anteaya: sorry for mistake | 17:05 |
| waynr | I still need up document the JJB 2.x demo tool's expected python module interface, actually there is much work to be done there | 17:05 |
| anteaya | cbader_: the channel gets busy with many conversations at once, pastes make things easier, thank you | 17:06 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on user profile pic url: https://review.openstack.org/267038 | 17:06 |
| anteaya | waynr: thank you for sharing, I hope you had an enjoyable presentation experience | 17:06 |
| cbader_ | anteaya: thank you for help | 17:06 |
| anteaya | cbader_: you're welcome | 17:06 |
| waynr | anteaya: it was great! lots of good questions, and I even plugged openstack-infra's zuul-based CI system when discussing the history of JJB | 17:07 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Improve "add card" UI https://review.openstack.org/267002 | 17:07 |
| anteaya | waynr: oh thank you | 17:07 |
| waynr | (in case it's not obvious I am a fan the openstack CI system) | 17:07 |
| anteaya | waynr: I'm glad you had an attentive audience | 17:07 |
| anteaya | waynr: always nice to state the obvious :) | 17:07 |
| mordred | waynr: \o/ | 17:08 |
| *** jistr has quit IRC | 17:08 | |
| anteaya | jeblair fungi pleia2 clarkb mordred so thoughts on adding ci-backup to cacti? | 17:08 |
| clarkb | greghaynes: looks like the nodepool stack is all +1 now by jenkins. Are you/we going to have time today to do this is the question | 17:08 |
| clarkb | greghaynes: I really really want to get it merged but so many things to do | 17:09 |
| anteaya | I'll add it if we have no reason for it to not be in cacti | 17:09 |
| * mordred also wants that merged | 17:09 | |
| mordred | anteaya: I do not know of any reasons - but jeblair or clarkb may have one | 17:09 |
| anteaya | mordred: thank you | 17:09 |
| clarkb | hrm, I don't think there is a reason not too, since that is just snmp | 17:09 |
| * anteaya waits for more input while making soup for lunch | 17:10 | |
| anteaya | clarkb: thank you | 17:10 |
| clarkb | but I know we have been cautious with the backup hosts | 17:10 |
| anteaya | makes sense | 17:10 |
| *** bardia has quit IRC | 17:10 | |
| greghaynes | clarkb: I am free and clear (as much as you are due to housing) | 17:10 |
| jeblair | it's probably worth the very minor risk | 17:10 |
| *** annegentle has quit IRC | 17:10 | |
| *** sfinucan has quit IRC | 17:10 | |
| *** zeih has quit IRC | 17:11 | |
| *** amotoki has quit IRC | 17:11 | |
| jeblair | and actually, afaict, we're already running it and have the port open, so we're taking the 'risk' without getting the benefit :) | 17:11 |
| zxiiro | curious, what size VMs does openstack use for Jenkins instances? | 17:11 |
| jeblair | anteaya: want to add it to the list in system-config? | 17:11 |
| anteaya | jeblair: I do, thank you | 17:11 |
| fungi | yeah, i see no real reason not to be monitoring the backup host(s) in cacti | 17:11 |
| clarkb | zxiiro: the jenkins masters are 8vcpu 30GB ram iirc | 17:11 |
| * anteaya figures out how to offer that patch | 17:11 | |
| clarkb | zxiiro: you can see them on http://cacti.openstack.org | 17:12 |
| zxiiro | clarkb: thanks | 17:12 |
| clarkb | greghaynes: ok, why don't I ping you after I get back from the bank and we can try then | 17:12 |
| jeblair | /dev/mapper/main-backups 2.0T 1002G 1015G 50% /opt/backups | 17:12 |
| *** apoorvad has joined #openstack-infra | 17:12 | |
| pleia2 | woo, new 2T backups overloard | 17:12 |
| pleia2 | overlord too | 17:12 |
| clarkb | there is no easy method for rolling old bcakups off with bup right? | 17:13 |
| jeblair | clarkb: s/easy// | 17:13 |
| jeblair | clarkb: well, i mean, they suggest starting a new backup tree | 17:13 |
| clarkb | ah ok | 17:13 |
| clarkb | so you sort of manage that atop bup | 17:14 |
| jeblair | clarkb: so i mean, we could do a thing where we rotate trees by date or something | 17:14 |
| mordred | jeblair: AFS? | 17:14 |
| openstackgerrit | Guido Günther proposed openstack/python-jenkins: Add support for handling promotions https://review.openstack.org/258032 | 17:14 |
| jeblair | mordred: nicely done | 17:14 |
| * mordred wins today's "suggest AFS as the solution to a problem" game | 17:15 | |
| clarkb | this is going to turn into a drinking game | 17:15 |
| jeblair | mordred is already winning that too | 17:15 |
| mordred | clarkb: it's not already a drinking game? | 17:15 |
| clarkb | apparently I am just behind on the drinking :) | 17:16 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Add non-voting Django 1.9 test to Horizon https://review.openstack.org/267043 | 17:17 |
| zaro | morning | 17:17 |
| *** esikachev has quit IRC | 17:18 | |
| anteaya | morning zaro | 17:18 |
| *** GheRiver1 has quit IRC | 17:19 | |
| clarkb | greghaynes: maybe if you get a chance you can skim through the stack and make sure it looks good to you | 17:19 |
| *** dizquierdo has quit IRC | 17:21 | |
| *** flepied has quit IRC | 17:22 | |
| *** GheRiver1 has joined #openstack-infra | 17:23 | |
| *** krtaylor has quit IRC | 17:25 | |
| clarkb | hogepodge: the state department has really good travel info for all the places as it related to USA passport carriers fwiw | 17:26 |
| fungi | jeblair: thanks again | 17:26 |
| *** krtaylor has joined #openstack-infra | 17:26 | |
| clarkb | er wrong window | 17:26 |
| *** mudassirlatif has joined #openstack-infra | 17:26 | |
| clarkb | but hey useful to everyone :) | 17:26 |
| fungi | anteaya: lmk when you post the cacti addition change and i'll fasttrack it | 17:26 |
| openstackgerrit | Merged openstack-infra/storyboard: Speed up story GET requests https://review.openstack.org/266961 | 17:26 |
| clarkb | (as not all the data is MURICA specific) | 17:26 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config: Add ci-backup.openstack.org to cacti https://review.openstack.org/267050 | 17:26 |
| anteaya | fungi: thank you | 17:26 |
| anteaya | there it is, not sure if I have the name of the server correct | 17:27 |
| anteaya | will respin if I guessed wrong | 17:27 |
| clarkb | anteaya: the name is documented on the ci docs | 17:27 |
| anteaya | clarkb: I found that useful thank you | 17:27 |
| anteaya | clarkb: pleia2 found this for me: http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n832 | 17:28 |
| *** _nadya_ has quit IRC | 17:28 | |
| *** markus_z has quit IRC | 17:28 | |
| anteaya | but according to http://docs.openstack.org/infra/system-config/sysadmin.html#backups | 17:28 |
| *** ayoung has quit IRC | 17:28 | |
| anteaya | it should be ci-backup-rs-ord.openstack.org | 17:28 |
| clarkb | anteaya: http://docs.openstack.org/infra/system-config/sysadmin.html#backups ya that | 17:28 |
| anteaya | so I can change it | 17:28 |
| * anteaya respins | 17:29 | |
| *** HeOS has quit IRC | 17:29 | |
| pabelanger | jeblair: so, thinking of toggling the color of the singlestat panel for the pipelines (check). Looking now http://grafana.openstack.org/dashboard/db/zuul-status we have 300+, to me that would be a good limit to indicate medium backlog | 17:29 |
| pabelanger | maybe 250 | 17:29 |
| *** xyang1 has quit IRC | 17:31 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config: Add ci-backup-rs-ord.openstack.org to cacti https://review.openstack.org/267050 | 17:31 |
| pabelanger | anteaya: jeblair: AJaeger: maybe we can merge https://review.openstack.org/#/c/264994/ today too. Demoing template support in grafana | 17:31 |
| *** aopincar_ has quit IRC | 17:31 | |
| anteaya | fungi clarkb ^^ | 17:31 |
| jeblair | pabelanger: before we do that, maybe we can decide whether the value is correct | 17:31 |
| *** Swami has quit IRC | 17:31 | |
| pabelanger | jeblair: true | 17:31 |
| pabelanger | jeblair: I only see 219 on status.o.o/zuul | 17:32 |
| jeblair | pabelanger: could you look into whether that value is the number of live changes or the number of both live and non-live ones? | 17:32 |
| *** mtanino has joined #openstack-infra | 17:32 | |
| *** fesp has joined #openstack-infra | 17:32 | |
| pabelanger | jeblair: sure | 17:32 |
| fungi | thanks anteaya | 17:33 |
| *** annegentle has joined #openstack-infra | 17:33 | |
| anteaya | fungi: thank you | 17:33 |
| * anteaya goes to have lunch | 17:33 | |
| *** xyang1 has joined #openstack-infra | 17:33 | |
| *** yamamoto has quit IRC | 17:34 | |
| *** fesp has quit IRC | 17:34 | |
| pabelanger | jeblair: we are using the current value of stats.gauges.zuul.pipeline.check.current_changes | 17:35 |
| *** kzaitsev_mb has quit IRC | 17:35 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Updates PyKMIP version https://review.openstack.org/267056 | 17:35 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show Project Groups on search page https://review.openstack.org/267027 | 17:35 |
| clarkb | pabelanger: that is the same value the sparkline on the zuul status page uses | 17:35 |
| jeblair | pabelanger: what i'm asking you to do is to look into that value and see if it represents what we want it to | 17:36 |
| clarkb | but the (226) on status.o.o/zuul is calcualted from the json iirc | 17:36 |
| pabelanger | clarkb: right, for some reason we have some driff between the grafana and it | 17:36 |
| clarkb | pabelanger: well I don't think there is a diff, since the number next to it is not from graphite | 17:36 |
| *** bcourt has quit IRC | 17:36 | |
| jeblair | pabelanger: i suspect that one or both of these may include non-live changes, and we should think about changing that | 17:36 |
| pabelanger | clarkb: Oh, that might be it | 17:36 |
| pabelanger | jeblair: okay, let me dig into it | 17:37 |
| jeblair | pabelanger: thanks | 17:37 |
| jeblair | anteaya: there was a point where gerrit was not rotating its logs, right? | 17:37 |
| jeblair | oh lunch | 17:38 |
| *** rlandy has quit IRC | 17:38 | |
| jeblair | fungi, clarkb: ^? | 17:38 |
| pleia2 | yeah | 17:38 |
| pleia2 | and it was filling up the disk | 17:38 |
| jeblair | pleia2: 2015-03-21-053701 appears to be our last gerrit backup | 17:38 |
| jeblair | i'm extracting it, and so far so good, though it is pulling out a _lot_ of gerrit logs | 17:39 |
| pleia2 | :\ | 17:39 |
| pleia2 | ah, yeah | 17:39 |
| jeblair | so i'm guessing/hoping that's before we implemented the rotation | 17:39 |
| pleia2 | I think it was over the summer that we did the log thing | 17:39 |
| * pleia2 nods | 17:39 | |
| jeblair | pleia2: cool, that timeframe works then | 17:39 |
| fungi | that timeline sounds right | 17:39 |
| *** rlandy has joined #openstack-infra | 17:39 | |
| jeblair | /home/gerrit2/review_site/tmp/gerrit_7014097362342183216_app/ | 17:40 |
| jeblair | did we do something about having lots of those ^ directories? | 17:40 |
| clarkb | I don't recall doing anything for lots of tmp dirs | 17:41 |
| jeblair | i guess not, though it looks like it's not huge... | 17:41 |
| pleia2 | nope (just checked on the gerrit server) | 17:41 |
| clarkb | jeblair: is that for melody? I think melody uses TMPDIR | 17:41 |
| jeblair | clarkb: i dunno? it has gerrit.war unpacked and lots of jars | 17:41 |
| pleia2 | 19M on the server, looks like a monthly rotation | 17:41 |
| jeblair | of gerrit dependencies | 17:41 |
| clarkb | oh then no I don't think that is melody | 17:41 |
| clarkb | melody uses tmp dirs to store the pngs | 17:41 |
| pleia2 | actually, some other cadence for the .jar files | 17:42 |
| jeblair | clarkb: i think that's in /tmp/javamelody | 17:42 |
| jeblair | pleia2: maybe each time we upgrade or something? | 17:42 |
| pleia2 | jeblair: too many for that, restarts maybe? | 17:42 |
| jeblair | pleia2: oh maybe | 17:43 |
| jeblair | pleia2: i restarted monday i think? | 17:43 |
| pleia2 | -rw-r--r-- 1 gerrit2 gerrit2 7.4K Dec 22 14:43 plugin_singleusergroup_151222_1443_889258577487934898.jar | 17:43 |
| pleia2 | -rw-r--r-- 1 gerrit2 gerrit2 7.4K Jan 11 22:05 plugin_singleusergroup_160111_2205_573377469593721012.jar | 17:43 |
| pleia2 | those look like restart dates ^^ | 17:43 |
| jeblair | pleia2: agreed | 17:43 |
| jeblair | zaro: do you know anything about gerrit unpacking things into /home/gerrit2/review_site/tmp/*/ every time it starts, and then not cleaning them up? | 17:44 |
| fungi | i think we have something unpacking the default included plugins? | 17:46 |
| fungi | i don't recall the exact details | 17:46 |
| zaro | jeblair: not the top of my head, but i think i saw something about this in cthe configurations doc. | 17:46 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient https://review.openstack.org/266955 | 17:46 |
| clarkb | cacti fo rthe backup server is approved | 17:46 |
| *** vgridnev has joined #openstack-infra | 17:46 | |
| *** Sukhdev has quit IRC | 17:47 | |
| zaro | fungi: not the same thing. unpacking the plugins is our way to install plugins and it doesn't go into tmp folder | 17:47 |
| pleia2 | thanks anteaya! | 17:47 |
| *** unicell has quit IRC | 17:47 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit https://review.openstack.org/267066 | 17:49 |
| *** bcourt has joined #openstack-infra | 17:50 | |
| pabelanger | jeblair: I suspect you are right. If I am reading this correct, formatStatusJSON() from the pipeline model is limiting the changes it is returning. While statsd seems to be logging all items in the queue | 17:50 |
| *** harlowja_at_home has quit IRC | 17:52 | |
| *** tzn has quit IRC | 17:52 | |
| *** _et_ has quit IRC | 17:53 | |
| *** yamamoto has joined #openstack-infra | 17:53 | |
| *** electrofelix has quit IRC | 17:53 | |
| *** tzn has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on user profile pic url: https://review.openstack.org/267038 | 17:54 |
| *** sridhar_ram has quit IRC | 17:54 | |
| anteaya | pleia2: thanks for your help | 17:55 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit https://review.openstack.org/267066 | 17:55 |
| *** shardy has quit IRC | 17:55 | |
| *** jordanP has quit IRC | 17:55 | |
| *** kushal has quit IRC | 17:57 | |
| *** amitgandhinz has joined #openstack-infra | 17:57 | |
| nibalizer | good morning | 17:57 |
| anteaya | morning nibalizer | 17:58 |
| nibalizer | i am traveling today so wont be around much | 17:58 |
| *** yamamoto has quit IRC | 17:58 | |
| *** tzn has quit IRC | 17:58 | |
| anteaya | jeblair: back now, reading backscroll looks like you found the answers you needed, I agree with what was stated re: gerrit logs | 17:58 |
| anteaya | nibalizer: safe travels, thanks for telling us | 17:58 |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project https://review.openstack.org/266674 | 17:58 |
| *** sabeen1 has joined #openstack-infra | 17:59 | |
| *** kushal has joined #openstack-infra | 17:59 | |
| *** rkukura has quit IRC | 18:00 | |
| *** mgould has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:01 | |
| greghaynes | clarkb: ohai, sorry, had to be on the phone with bank for waaaay too long | 18:01 |
| greghaynes | clarkb: and sounds like you are doing the same but in person ;) - I checked over the nodepool stack last night and it looked good to me | 18:02 |
| *** [1]Thelo has joined #openstack-infra | 18:02 | |
| yolanda | hi greghaynes, thanks for your review on dib lvm. I fixed the issues you noted | 18:02 |
| greghaynes | yolanda: ok, awesome | 18:02 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Improve "add card" UI https://review.openstack.org/267002 | 18:03 |
| yolanda | care taking another look? and also to https://review.openstack.org/259881, that is partitioning elemetn | 18:03 |
| greghaynes | yolanda: Ya, I'll check it out in a bit. | 18:03 |
| *** yaume has quit IRC | 18:03 | |
| yolanda | thanks! | 18:03 |
| *** e0ne has quit IRC | 18:04 | |
| *** sabeen3 has joined #openstack-infra | 18:04 | |
| zaro | jeblair, fungi: ok. here's what the tmp folder is for https://groups.google.com/forum/#!searchin/repo-discuss/tmp/repo-discuss/Z1STG5t3v6E/8T-i7Ucl_FwJ | 18:04 |
| *** flepied has joined #openstack-infra | 18:04 | |
| *** Thelo has quit IRC | 18:04 | |
| *** [1]Thelo is now known as Thelo | 18:04 | |
| *** Swami has joined #openstack-infra | 18:05 | |
| zaro | looks like cleanup has not been fixed | 18:05 |
| jeblair | yeah | 18:05 |
| anteaya | dims_: do you want to address mriedem's comment on this patch of yours I merged yesterday? https://review.openstack.org/#/c/266119/4 | 18:06 |
| *** treaki_ has quit IRC | 18:06 | |
| anteaya | dims_: we seem to have overlooked an important leading comma | 18:06 |
| mriedem | anteaya: dims_: i'm fixing it | 18:06 |
| *** sabeen1 has quit IRC | 18:06 | |
| fungi | zaro: jeblair: i guess it should be safe to have the initscript wipe them if we care? | 18:06 |
| anteaya | mriedem: okey dokey, tell me when you have a patch up and I'll review, sorry about that | 18:06 |
| dims_ | mriedem : thanks! | 18:07 |
| fungi | zaro: though perhaps this is a regression. it looked like those fixes linked in the last comment merged years ago | 18:08 |
| *** rajinir has quit IRC | 18:08 | |
| *** ganso has quit IRC | 18:08 | |
| *** boris-42 has quit IRC | 18:08 | |
| *** erlon has quit IRC | 18:08 | |
| *** rguillebert has quit IRC | 18:08 | |
| *** samyewell has quit IRC | 18:08 | |
| *** larainema has quit IRC | 18:08 | |
| *** jed56 has quit IRC | 18:08 | |
| *** yonglihe has quit IRC | 18:08 | |
| *** Ryan_Lane has quit IRC | 18:08 | |
| *** kgalanov has quit IRC | 18:08 | |
| *** igorbelikov has quit IRC | 18:08 | |
| *** dtardivel has quit IRC | 18:08 | |
| *** maximov has quit IRC | 18:08 | |
| *** briancurtin has quit IRC | 18:08 | |
| *** dougwig has quit IRC | 18:08 | |
| *** zhenguo has quit IRC | 18:08 | |
| *** xgerman has quit IRC | 18:08 | |
| *** mwhahaha has quit IRC | 18:08 | |
| *** auggy has quit IRC | 18:08 | |
| *** GheRivero has quit IRC | 18:08 | |
| *** skath_ has quit IRC | 18:08 | |
| *** patrickeast has quit IRC | 18:08 | |
| *** zxiiro has quit IRC | 18:08 | |
| *** sweston has quit IRC | 18:08 | |
| *** othermordred has quit IRC | 18:08 | |
| *** dham1 has quit IRC | 18:08 | |
| *** ams_ has quit IRC | 18:08 | |
| *** teran has quit IRC | 18:08 | |
| *** monester has quit IRC | 18:08 | |
| *** sballe has quit IRC | 18:08 | |
| *** ashtokolov has quit IRC | 18:08 | |
| *** ramishra_ has quit IRC | 18:08 | |
| *** tpeoples has quit IRC | 18:08 | |
| *** etoews has quit IRC | 18:08 | |
| *** ameade has quit IRC | 18:08 | |
| *** jraim has quit IRC | 18:08 | |
| *** fmccrthy has quit IRC | 18:08 | |
| *** DuncanT has quit IRC | 18:08 | |
| *** serverascode has quit IRC | 18:08 | |
| *** betherly has quit IRC | 18:08 | |
| *** mgkwill has quit IRC | 18:08 | |
| *** zhiyan has quit IRC | 18:08 | |
| *** evgenyl has quit IRC | 18:08 | |
| *** lindsayk_ has quit IRC | 18:08 | |
| *** rackertom has quit IRC | 18:08 | |
| *** kozhukalov has quit IRC | 18:08 | |
| *** simonmcc has quit IRC | 18:08 | |
| *** electrical has quit IRC | 18:08 | |
| *** edwarnicke has quit IRC | 18:08 | |
| *** warpc has quit IRC | 18:08 | |
| *** Ng has quit IRC | 18:08 | |
| *** fdegir has quit IRC | 18:08 | |
| zaro | fungi: it looks like a regression to me. i'll enter an issue for it and see if i can get confirmation. | 18:10 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Fix ENABLED_SERVICES concat failure for stable branch jobs https://review.openstack.org/267076 | 18:10 |
| mriedem | anteaya: dims_: sdague: ^ | 18:10 |
| mtreinish | woah, someone must not like irccloud :) | 18:10 |
| *** tonytan4ever has joined #openstack-infra | 18:10 | |
| *** rackertom has joined #openstack-infra | 18:10 | |
| *** ams_ has joined #openstack-infra | 18:11 | |
| *** GheRivero has joined #openstack-infra | 18:11 | |
| anteaya | mriedem: +2 | 18:12 |
| *** jsavak has quit IRC | 18:12 | |
| mriedem | don't all +W at once now | 18:12 |
| anteaya | mtreinish: I'm missing the context | 18:12 |
| *** jsavak has joined #openstack-infra | 18:12 | |
| sdague | mriedem: +A | 18:12 |
| fungi | anteaya: alt+= | 18:12 |
| fungi | look at all the kline quits | 18:12 |
| anteaya | ah | 18:12 |
| fungi | one of the freenode operators just blocked a popular bouncer site i guess | 18:13 |
| Clint | it happens | 18:13 |
| nibalizer | wow | 18:13 |
| nibalizer | any idea why? | 18:14 |
| *** aopincar_ has joined #openstack-infra | 18:14 | |
| fungi | no mention of anything on their staff twitter, so you'd probably need to ask in #freenode | 18:14 |
| *** serverascode has joined #openstack-infra | 18:14 | |
| jeblair | 18:13 < k6ka> Thero: k-line was accidental. collateral damage whilst trying to stop a troll. | 18:14 |
| *** DuncanT has joined #openstack-infra | 18:14 | |
| jeblair | i lurk in #freenode | 18:14 |
| anteaya | did they stop the troll? | 18:15 |
| fungi | i should do that, thanks for the details | 18:15 |
| *** ccrouch has quit IRC | 18:15 | |
| fungi | does anyone ever stop a troll? at best you slow them down while they look for a new way to come back and annoy people | 18:15 |
| anteaya | I mean if they took out so many others did they at least achieve their goal? | 18:15 |
| *** edwarnicke has joined #openstack-infra | 18:15 | |
| anteaya | ha ha ha | 18:15 |
| anteaya | true | 18:15 |
| fungi | eventually you just make it cumbersome enough for them that they hopefully move on to something less tedious | 18:15 |
| jeblair | they even updated their channel topic: "disruption to irccloud unintended - our apologies, it should be fixed now!" | 18:15 |
| anteaya | that was kind of them | 18:16 |
| *** mwhahaha_ has joined #openstack-infra | 18:16 | |
| *** electrical has joined #openstack-infra | 18:16 | |
| *** warpc has joined #openstack-infra | 18:16 | |
| *** sweston has joined #openstack-infra | 18:16 | |
| *** mwhahaha_ is now known as mwhahaha | 18:16 | |
| *** sridhar_ram has joined #openstack-infra | 18:16 | |
| *** mgkwill has joined #openstack-infra | 18:17 | |
| anteaya | like a cattle gate | 18:17 |
| jeblair | fungi: i don't highly recommend lurking in #freenode -- the highlight spam is near-continuous :) | 18:17 |
| *** unicell has joined #openstack-infra | 18:17 | |
| *** jraim has joined #openstack-infra | 18:17 | |
| *** ccrouch has joined #openstack-infra | 18:18 | |
| fungi | oh, that's unfortunate | 18:18 |
| *** larainema has joined #openstack-infra | 18:18 | |
| *** kozhukalov has joined #openstack-infra | 18:18 | |
| fungi | jeblair: on the other hand, i can just tell weechat not to highlight matches in that channel. quite simple to configure actually | 18:18 |
| *** ccrouch has quit IRC | 18:18 | |
| *** ramishra_ has joined #openstack-infra | 18:18 | |
| *** ccrouch has joined #openstack-infra | 18:18 | |
| jeblair | there you go | 18:18 |
| *** gyee has joined #openstack-infra | 18:18 | |
| anteaya | they have cattle gates around university of toronto buildings in the middle of the city | 18:19 |
| *** simonmcc has joined #openstack-infra | 18:19 | |
| fungi | i already have activity and highlight levels tuned differently for different channels, to help me prioritize and cope with my terribly slow reading speed | 18:19 |
| anteaya | since when the buildings were built there were cattle all around | 18:19 |
| *** NikolaTesla has joined #openstack-infra | 18:19 | |
| *** Ng has joined #openstack-infra | 18:20 | |
| fungi | anteaya: that's pretty awesome. more cities need cattle grates | 18:20 |
| *** lindsayk has quit IRC | 18:20 | |
| anteaya | agreed | 18:20 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove nova-commit-message job https://review.openstack.org/259569 | 18:20 |
| *** piet has quit IRC | 18:20 | |
| *** teran has joined #openstack-infra | 18:21 | |
| *** VivekVC is now known as Vivek | 18:21 | |
| anteaya | I think they keep them for historical reasons now, they are really pretty wrought iron affairs | 18:21 |
| *** etoews has joined #openstack-infra | 18:21 | |
| *** ameade has joined #openstack-infra | 18:21 | |
| *** Vivek has quit IRC | 18:21 | |
| *** Vivek has joined #openstack-infra | 18:21 | |
| *** sballe has joined #openstack-infra | 18:21 | |
| *** evgenyl has joined #openstack-infra | 18:21 | |
| *** lindsayk has joined #openstack-infra | 18:21 | |
| *** erlon has joined #openstack-infra | 18:22 | |
| *** amuller_afk is now known as amuller | 18:22 | |
| *** monester has joined #openstack-infra | 18:22 | |
| *** yonglihe has joined #openstack-infra | 18:22 | |
| *** rkukura has joined #openstack-infra | 18:22 | |
| anteaya | sdague: you let me know if you line up any support for 259569 and I too will support it | 18:22 |
| *** zxiiro has joined #openstack-infra | 18:22 | |
| *** maximov has joined #openstack-infra | 18:23 | |
| *** betherly has joined #openstack-infra | 18:23 | |
| sdague | anteaya: well I feel like we got there on the thread | 18:23 |
| *** samyewell has joined #openstack-infra | 18:23 | |
| * anteaya grimaces at the thought of wading through the thread | 18:23 | |
| fungi | mtreinish: not sure if you saw earlier, but daily catch-up pruning of the subunit2sql database is probably around 15 minutes | 18:23 |
| *** patrickeast has joined #openstack-infra | 18:23 | |
| fungi | not too terrible | 18:23 |
| anteaya | is there a post from lana that says she agrees? | 18:23 |
| *** ganso has joined #openstack-infra | 18:23 | |
| *** skath_ has joined #openstack-infra | 18:23 | |
| *** othermordred has joined #openstack-infra | 18:23 | |
| *** zhiyan has joined #openstack-infra | 18:24 | |
| AJaeger | let's wait for loquacities (that's lana's nick) to wake up and discuss with her | 18:24 |
| fungi | anteaya: that last thread it sounded like she was fine with the new direction, but yeah she can always confirm explicitly in here | 18:24 |
| *** tpeoples has joined #openstack-infra | 18:24 | |
| fungi | er, in that last reply to the thread | 18:24 |
| mtreinish | fungi: yeah I did catch that. It's pretty good | 18:24 |
| fungi | she'll likely be around any time now | 18:24 |
| *** lindsayk_ has joined #openstack-infra | 18:25 | |
| *** Jeffrey4l has quit IRC | 18:25 | |
| mtreinish | I did set the cron job up to run weekly: https://review.openstack.org/176564 | 18:25 |
| anteaya | well in http://lists.openstack.org/pipermail/openstack-dev/2016-January/083806.html it sounds like she wants the defcore ptl to weigh in | 18:25 |
| mtreinish | but I can change that to be daily if you think that's better | 18:25 |
| *** ericksonsantos has quit IRC | 18:25 | |
| *** xgerman has joined #openstack-infra | 18:25 | |
| anteaya | so yeah, I'll wait a bit, thanks | 18:25 |
| fungi | mtreinish: should be fine. maybe an hour to complete weekly, but likely less | 18:25 |
| *** esikachev has joined #openstack-infra | 18:25 | |
| *** briancurtin has joined #openstack-infra | 18:25 | |
| *** dougwig has joined #openstack-infra | 18:25 | |
| *** patrickeast_ has joined #openstack-infra | 18:26 | |
| *** rguillebert has joined #openstack-infra | 18:26 | |
| *** bardia has joined #openstack-infra | 18:26 | |
| mtreinish | and that first run will likely be much slower it's got another 6months of data sitting there still | 18:26 |
| anteaya | would the defcore ptl be eglute? | 18:26 |
| *** dham1 has joined #openstack-infra | 18:26 | |
| *** patrickeast_ has quit IRC | 18:26 | |
| anteaya | defcore-release is empty: https://review.openstack.org/#/admin/groups/627,members | 18:26 |
| mtreinish | anteaya: there's no such thing, the reference was for projects covered by defcore's ptl | 18:27 |
| *** fdegir has joined #openstack-infra | 18:27 | |
| mtreinish | anteaya: defcore is a board thing so ptl doesn't make sense for that | 18:27 |
| anteaya | mtreinish: what does that entail? | 18:27 |
| *** ashtokolov has joined #openstack-infra | 18:27 | |
| *** HeOS has joined #openstack-infra | 18:27 | |
| mtreinish | anteaya: nova, glance, keystone, etc.. | 18:27 |
| *** Ryan_Lane has joined #openstack-infra | 18:27 | |
| anteaya | I'm just trying to figure out who to get to review sdague's patch to avoid people coming back to me saying I didn't consult them | 18:28 |
| anteaya | now I'm fine with people thinking I made the wrong decision or a made a mistake | 18:28 |
| *** ociuhandu has quit IRC | 18:28 | |
| anteaya | but I do like folks to at least have a heads up, so trying to do that | 18:28 |
| *** _nadya_ has joined #openstack-infra | 18:29 | |
| *** dtardivel has joined #openstack-infra | 18:29 | |
| *** auggy has joined #openstack-infra | 18:29 | |
| *** jed56 has joined #openstack-infra | 18:29 | |
| anteaya | or we can just move ahead with out me | 18:29 |
| anteaya | that is an option | 18:29 |
| *** _nadya_ has quit IRC | 18:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Constraints jobs non-voting for neutron-vpnaas https://review.openstack.org/265807 | 18:30 |
| *** _nadya_ has joined #openstack-infra | 18:30 | |
| *** dims_ has quit IRC | 18:30 | |
| *** mgagne has quit IRC | 18:30 | |
| *** rcernin has joined #openstack-infra | 18:30 | |
| *** trown is now known as trown|lunch | 18:31 | |
| *** rajinir has joined #openstack-infra | 18:31 | |
| *** mgagne has joined #openstack-infra | 18:31 | |
| *** kgalanov has joined #openstack-infra | 18:31 | |
| *** mgagne is now known as Guest82803 | 18:31 | |
| *** dims has joined #openstack-infra | 18:32 | |
| *** igorbelikov has joined #openstack-infra | 18:32 | |
| *** fmccrthy has joined #openstack-infra | 18:32 | |
| *** Guest82803 is now known as mgagne | 18:32 | |
| *** kdas_ has joined #openstack-infra | 18:33 | |
| *** mgagne has quit IRC | 18:33 | |
| *** mgagne has joined #openstack-infra | 18:33 | |
| *** otter768 has joined #openstack-infra | 18:33 | |
| *** ayoung has joined #openstack-infra | 18:33 | |
| greghaynes | clarkb: hrm, pep8 fail on https://review.openstack.org/#/c/266086/ (dib prune) | 18:33 |
| greghaynes | ianw: ^ | 18:33 |
| clarkb | is it running bashate? | 18:34 |
| greghaynes | we should rename that job to just be "lint" since it doen't do much pep8 | 18:34 |
| greghaynes | it runs dib-lint | 18:34 |
| clarkb | ah | 18:34 |
| greghaynes | and pep8 on some things | 18:34 |
| *** sridhar_ram has quit IRC | 18:35 | |
| greghaynes | oh actually, it is pep8 that failed. I wonder if that or something else updated... | 18:35 |
| *** sridhar_ram has joined #openstack-infra | 18:35 | |
| *** piet has joined #openstack-infra | 18:35 | |
| *** kushal has quit IRC | 18:35 | |
| *** yamahata has quit IRC | 18:35 | |
| *** woodster_ has joined #openstack-infra | 18:36 | |
| *** kdas_ has quit IRC | 18:36 | |
| *** zhenguo has joined #openstack-infra | 18:36 | |
| *** esikachev has quit IRC | 18:36 | |
| *** kushal has joined #openstack-infra | 18:37 | |
| *** otter768 has quit IRC | 18:38 | |
| *** e0ne has joined #openstack-infra | 18:38 | |
| *** treaki_ has joined #openstack-infra | 18:40 | |
| *** jaypipes has quit IRC | 18:41 | |
| odyssey4me | greghaynes yeah, pep8 released v1.7.0 today - so if you only have 'deps=flake8' as the deps in your tox config, then you'll have to do some python pep8 fixes | 18:42 |
| *** darvon has quit IRC | 18:43 | |
| *** ig0r_ has joined #openstack-infra | 18:43 | |
| greghaynes | :( | 18:43 |
| greghaynes | ok | 18:43 |
| *** tonytan4ever has quit IRC | 18:43 | |
| greghaynes | odyssey4me: ty | 18:43 |
| jeblair | anteaya: i'd ask loquacities to weigh in on it again too; her last comment was a -1, but i think further convergence happened on the thread after that. | 18:44 |
| *** jaypipes has joined #openstack-infra | 18:44 | |
| zaro | jeblair, fungi : i just verified that the gerrit tmp folder cleanup works in 2.11. the plugin files are probably left around from older versions of gerrit. if you manually cleanup the tmp folder and start/stop gerrit you'll see that tmp is empty. | 18:45 |
| *** darvon has joined #openstack-infra | 18:45 | |
| odyssey4me | greghaynes sure thing :) another option is to change it to 'deps = -r{toxinidir}/test-requirements.txt' and then have constraints in test-requirements.txt that you're comfortable with and maintain | 18:45 |
| jeblair | zaro: we have one from decemeber 22 | 18:45 |
| *** rkukura has quit IRC | 18:45 | |
| anteaya | jeblair: yes that seems to be what I'm seeing too, yeah I'll wait for her to be online and she what she says | 18:45 |
| greghaynes | odyssey4me: Ya, I mostly just wanted to make sure our CI wasnt hosed, looks like we just have one line that needs changing so not a big deal | 18:45 |
| *** bharathm has joined #openstack-infra | 18:45 | |
| odyssey4me | greghaynes happy days then :) | 18:46 |
| jeblair | zaro: but we can try that and keep an eye on it | 18:46 |
| *** _nadya_ has quit IRC | 18:46 | |
| *** rkukura has joined #openstack-infra | 18:48 | |
| *** kushal has quit IRC | 18:48 | |
| *** rcrit has joined #openstack-infra | 18:49 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix pep8 import not at top of file https://review.openstack.org/267094 | 18:49 |
| greghaynes | SpamapS: ianw ^ needed to unbreak DIB CI | 18:49 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors https://review.openstack.org/253521 | 18:49 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 18:49 |
| *** sridhar_ram1 has joined #openstack-infra | 18:50 | |
| SpamapS | greghaynes: thats fixed in the gentoo review | 18:50 |
| SpamapS | greghaynes: so you could just +A that | 18:50 |
| greghaynes | SpamapS: hrm | 18:50 |
| greghaynes | ;/ | 18:50 |
| greghaynes | er, :/ | 18:51 |
| SpamapS | and I'm ok with such a transgression ;) | 18:51 |
| *** ayoung has quit IRC | 18:51 | |
| *** rossella_s has quit IRC | 18:52 | |
| *** rossella_s has joined #openstack-infra | 18:52 | |
| *** sridhar_ram has quit IRC | 18:54 | |
| *** daemontool has quit IRC | 18:54 | |
| greghaynes | SpamapS: ok, done | 18:54 |
| greghaynes | w00t we support gentoo soon | 18:54 |
| *** daemontool has joined #openstack-infra | 18:54 | |
| zaro | jeblair: i just tested it on review-dev.o.o and tmp folder was cleaned up. | 18:55 |
| openstackgerrit | lifeless proposed openstack-dev/cookiecutter: Require tox 2.0 https://review.openstack.org/267099 | 18:56 |
| *** tonytan4ever has joined #openstack-infra | 18:57 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit https://review.openstack.org/267066 | 19:00 |
| *** peristeri has quit IRC | 19:00 | |
| *** yamahata has joined #openstack-infra | 19:01 | |
| *** ericksonsantos has joined #openstack-infra | 19:01 | |
| *** mriedem is now known as mriedem_meeting | 19:02 | |
| *** ybathia has joined #openstack-infra | 19:02 | |
| *** kushal has joined #openstack-infra | 19:02 | |
| *** ybathia_ has joined #openstack-infra | 19:03 | |
| lifeless | Nakato: when you're around https://review.openstack.org/#/c/267096/1 | 19:03 |
| lifeless | fungi: also you should look at https://review.openstack.org/#/c/267096/1 I think | 19:03 |
| openstackgerrit | Merged openstack/python-jenkins: Add support for handling promotions https://review.openstack.org/258032 | 19:03 |
| *** rkukura has quit IRC | 19:03 | |
| fungi | dhellmann: ttx: you're probably the two other main stakeholders in the arrangement of the reference/extra-atcs file format... would you have any objection to extending the reference/projects.yaml schema to hold that information instead? | 19:04 |
| *** jaosorior has quit IRC | 19:05 | |
| *** jaosorior has joined #openstack-infra | 19:05 | |
| eglute | anteaya how can I help? | 19:06 |
| anteaya | eglute: hi | 19:06 |
| anteaya | well I'm not sure you can | 19:06 |
| eglute | hello! | 19:06 |
| *** ybathia has quit IRC | 19:06 | |
| *** ybathia_ is now known as ybathia | 19:06 | |
| anteaya | we were discussing a patch | 19:06 |
| eglute | link? | 19:07 |
| anteaya | and I had thought you were referenced but I might be wrong | 19:07 |
| anteaya | let me get that | 19:07 |
| eglute | thank you | 19:07 |
| *** dtardivel has quit IRC | 19:07 | |
| anteaya | this is the mailing list thread: http://lists.openstack.org/pipermail/openstack-dev/2016-January/083806.html | 19:08 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/hacking: Do not call "rm -rf" on arbitrary user input https://review.openstack.org/267104 | 19:08 |
| fungi | lifeless: sdague: having trouble context-switching to wrap my head around the implications of 267096 at the moment, but at a minimum the constraints-specific tox jobs are designed to be integration tests between the requirements repo and the repo for the change being tested, whereas the normal tox jobs definitely are not, so this would require pivoting normal tox-based jobs for all repos for the | 19:08 |
| fungi | desired behavior to take effect | 19:08 |
| anteaya | eglute: this is the patch under discussion: https://review.openstack.org/#/c/259569/ | 19:08 |
| sdague | fungi: ok, so here is where we stand | 19:09 |
| krotscheck | Found link http://localhost/wheel/Ubuntu-14.04/astroid/astroid-1.3.8-py2.py3-none-any.whl | 19:10 |
| * krotscheck does a little dance | 19:10 | |
| sdague | I started doing the neutron style way, however, in the broader nova team we spent 3 years making a switch from run_tests.sh to tox, and spend a non trivial amount of time with contributors debugging the fact they are running tests incorrectly | 19:10 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient https://review.openstack.org/266955 | 19:10 |
| sdague | so lots of folks just wanted the current targets to do the right thing | 19:11 |
| sdague | fungi: to the best of my understanding, the only real difference between the jobs is the - zuul-git-prep-upper-constraints | 19:11 |
| *** rkukura has joined #openstack-infra | 19:12 | |
| eglute | anteaya you are right, I do not see that I can help, and it does not appear to affect defcore. Let me know if there is anything else! | 19:12 |
| sdague | so if we could add that to all the existing jobs, it would make the tox -e py27 targets all be able to work this way (for instance) | 19:12 |
| sdague | I guess that's the piece we need to get agreement | 19:12 |
| anteaya | eglute: thank you for taking a look, sorry for my mistake | 19:12 |
| fungi | sdague: lifeless: we likely will still need a dichotomy of "test with constraints because this project uses requirements" job definitions and "this project doesn't participate in openstack global requirements so don't use constraints" job definitions unless we add some more scripted tooling to match against the projects.txt too | 19:12 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-pip: Added extra-index-url to pip.conf https://review.openstack.org/267107 | 19:13 |
| eglute | anteaya no worries! I am usually around on IRC, happy to help | 19:13 |
| anteaya | eglute: thank you :) | 19:13 |
| sdague | fungi: whether or not to test with constraints is controlled project local in tox.ini | 19:13 |
| lifeless | sigh pyeclib | 19:13 |
| *** vivekd has joined #openstack-infra | 19:13 | |
| mordred | krotscheck: WOOT! | 19:13 |
| lifeless | mordred: I am going to guess its a missing package dep | 19:13 |
| sdague | as long as our infrastructure makes the constraints file always available, then I think we're done | 19:13 |
| mordred | lifeless: yah. | 19:13 |
| sdague | and projects opt in by changing their targets | 19:13 |
| mordred | notmyname: we have a pbr patch failing on a pyeclib thing - is there something special needed related to that? | 19:14 |
| lifeless | fungi: we use constraints for *everyone* in devstack jobs | 19:14 |
| lifeless | fungi: whether they participate in g-r or not | 19:14 |
| fungi | sdague: i'll have to take a look at nova's configuration to see whether it's already using zuul-cloner to get the right refs when run in our ci, but that will have to wait for tomorrow, so removing my -1 for now | 19:14 |
| lifeless | fungi: and for unit tests and so on constraints tells us what versions work with any openstack deps they have | 19:15 |
| notmyname | mordred: what's the bug? | 19:15 |
| fungi | under a tight deadline for getting the atc passes generation working | 19:15 |
| krotscheck | mordred: Indeed | 19:15 |
| mordred | notmyname: http://logs.openstack.org/48/263748/6/check/gate-pbr-installation-dsvm/7afe9d6/console.html#_2016-01-13_15_05_34_688 | 19:15 |
| lifeless | fungi: sdague: -> I don't think g-r participation is tightly coupled to constraints use | 19:16 |
| notmyname | mordred: ah yes | 19:16 |
| mordred | lifeless: I think we need to install liberasurecode-dev - but I'm not sure why it's not installed already for that job | 19:16 |
| fungi | lifeless: okay, that seems maybe workable then | 19:16 |
| fungi | lifeless: sdague: to make it work the way 267096 wants, any difference between the constraints and non-constraints job definitions probably needs to get copied into nova's tox.ini instead | 19:16 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 19:16 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh https://review.openstack.org/267117 | 19:16 |
| notmyname | mordred: as according to plan, recent pyeclib has stopped bundling up the liberasurecode library | 19:16 |
| mordred | notmyname: yay! | 19:16 |
| lifeless | mordred: remember we install them ourselves in integration.sh | 19:17 |
| notmyname | mordred: so that's a good thing, right, but it does mean that you need to install liberasurecode itself now | 19:17 |
| lifeless | fungi: thats back to running zuul-cloner inside tox, which I thought we had significant objections to | 19:17 |
| mordred | notmyname: cool. thanks! | 19:17 |
| jeblair | fungi: isn't it right in 267096? it does not run zuul-cloner, it just pip install from a file over https | 19:17 |
| krotscheck | Two questions. 1- Our puppet modules are installed from master, yes? 2- Should I be adding the wheel mirror location to /home/jenkins/.pydistutils.cfg as well? | 19:18 |
| jeblair | fungi: (i mean, isn't the answer to your question right there in 267096?) | 19:18 |
| lifeless | jeblair: if we do that, then depends-on and stable branches won't get the right constraints file | 19:18 |
| jeblair | lifeless: right | 19:18 |
| fungi | jeblair: probably | 19:18 |
| lifeless | jeblair: to that end run-tox already exports upper-constraints to be a local path | 19:18 |
| jeblair | lifeless: i'm saying i think fungi is prematurely lifting his objection | 19:18 |
| AJaeger | rbradfor: if you build the docs locally, does that work? | 19:18 |
| sdague | lifeless: we just have to set the branch in the url when we branch, right? | 19:19 |
| fungi | jeblair: i just don't want to leave an objection on a change unless i have time to look closely at it | 19:19 |
| lifeless | jeblair: ack | 19:19 |
| lifeless | sdague: that takes care of branches, not of depends-on | 19:19 |
| lifeless | sdague: and we need depends-on for punch-through jobs | 19:20 |
| sdague | lifeless: ok, explain what you mean there? | 19:20 |
| jeblair | fungi, lifeless, sdague: I have left a -1 on that change because i think it's fine to have an objection and ask that the proposer of a patch look into it even if i don't immediately have the time. | 19:20 |
| fungi | needs to respect $ZUUL_REF | 19:20 |
| fungi | thanks jeblair | 19:20 |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Split changelog on nulls instead of ( https://review.openstack.org/263748 | 19:20 |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Add libjpeg and liberasurecode for tests https://review.openstack.org/267121 | 19:20 |
| sdague | except, in the jobs zuul cloner builds and exports that file | 19:20 |
| mordred | lifeless: ^^ (also notmyname in case you care- thanks) | 19:20 |
| sdague | so overrides the url | 19:20 |
| dhellmann | fungi : I have no objection to moving atc data into the project files. What's the motivation for the change? | 19:20 |
| sdague | https://github.com/openstack-infra/project-config/blob/c122a1035e7f20d852f20bec396669e64839e867/jenkins/scripts/run-tox.sh#L121 | 19:21 |
| lifeless | mordred: hammer applied | 19:21 |
| fungi | dhellmann: repeatedly writing parsers for our ever changing file format rather than using a standard | 19:21 |
| mordred | lifeless: yup | 19:21 |
| mordred | lifeless: although we should be able to remove the hammer in our new bindep overlord world | 19:21 |
| openstackgerrit | Rob Crittenden proposed openstack-infra/devstack-gate: Add tls-proxy as an available service https://review.openstack.org/188436 | 19:21 |
| dhellmann | fungi : so you're just tired of doing the same thing over and over? sheesh. ;-) | 19:21 |
| lifeless | mordred: indeedy | 19:22 |
| fungi | dhellmann: also feeling icky whenever i for line in open('extra-atcs'): | 19:22 |
| sdague | lifeless: does that not cover the issue? | 19:22 |
| lifeless | sdague: it does for -constraints jobs, not the normal ones today. Because AIUI historical reasons | 19:22 |
| dhellmann | fungi : haha, yeah. let's move things over. it'll probably make the governance repo rendering easier anyway. Want me to take stab at it, or are you going to? | 19:22 |
| lifeless | sdague: I was working on a patch set to unify all that | 19:22 |
| sdague | lifeless: ok, where is that distinction? | 19:22 |
| fungi | dhellmann: and then the fact that we inconsistently refer to project team names between it and projects.yaml, so it adds a need for mapping code to combine the two into a singular dataset | 19:23 |
| lifeless | but it assploded in complexity | 19:23 |
| dhellmann | fungi : ew | 19:23 |
| *** ayoung has joined #openstack-infra | 19:23 | |
| lifeless | sdague: oh, my memory - I was working on neutron's stadium needs, which is different again. https://review.openstack.org/#/c/252680/ | 19:23 |
| fungi | dhellmann: for right now i'm just going to parse it and try to magically map it up to projects.yaml, but i'm happy to submit something in the future to simplify this for all of us | 19:23 |
| sdague | right, we don't need all of that | 19:24 |
| lifeless | sdague: the distinction - I'm fuzzy on it - but my understanding is that zuul-cloner is new, and most of the jobs used some other mechanism to get code on disk | 19:24 |
| sdague | ok, but we're voting with at least some of them right? | 19:24 |
| lifeless | sdague: ECONFUSED | 19:24 |
| mordred | lifeless: all jobs _should_ be using zuul cloner to get git repos on disk | 19:24 |
| *** peristeri has joined #openstack-infra | 19:25 | |
| lifeless | mordred: thats my understanding yes | 19:25 |
| sdague | ok, so the change I want to make | 19:25 |
| dougwig | all neutron stadium jobs that i'm aware of are using zuul-cloner now. | 19:25 |
| mordred | \o/ | 19:25 |
| sdague | make gerrit-git-prep == zuul-git-prep-upper-constraints | 19:25 |
| fungi | i have a couple of builder macros that seem to be effective at replacing gerrit-git-prep and friends with zuul-cloner logic | 19:25 |
| sdague | so we always have upper constraints available | 19:25 |
| lifeless | dougwig: via a tox install override though, which is not the same thing at all :) | 19:25 |
| *** piet has quit IRC | 19:26 | |
| dougwig | lifeless: oh, indeed, i like your change. | 19:26 |
| sdague | then I think this all works | 19:26 |
| AJaeger | rbradfor: are you still around? | 19:26 |
| *** aopincar_ has quit IRC | 19:26 | |
| jeblair | sdague: and alder the pip install command in 267096 to use the zuul-cloner installed constraints file? | 19:27 |
| sdague | because otherwise neutron wouldn't be working | 19:27 |
| jeblair | alter | 19:27 |
| sdague | jeblair: no | 19:27 |
| *** Sukhdev has joined #openstack-infra | 19:27 | |
| sdague | because there is an env override | 19:27 |
| sdague | which is set by run-tox.sh | 19:27 |
| sdague | https://github.com/openstack-infra/project-config/blob/c122a1035e7f20d852f20bec396669e64839e867/jenkins/scripts/run-tox.sh#L121 | 19:27 |
| sdague | which is how the infrastructure tells us what to run | 19:27 |
| sdague | when running in the infrastructure | 19:27 |
| *** rockyg has joined #openstack-infra | 19:27 | |
| sdague | right? | 19:27 |
| *** piet has joined #openstack-infra | 19:28 | |
| *** _nadya_ has joined #openstack-infra | 19:28 | |
| jeblair | sdague: ah, so 267096 will use the env overriden constraints file which is the zuul-cloner installed constraints file; local non-zuul usage would use the internet fetched one? | 19:28 |
| sdague | because https://github.com/openstack/nova/blob/1005478e03b7da8cedc163ea4de5ffa5b4d6dbac/tox.ini#L12 | 19:28 |
| sdague | I believe that's what that line means | 19:28 |
| AJaeger | rbradfor: the files are on docs.openstack.org and have proper permissions. I do not know why you get a forbidden, might be some configuration on the host - and I can't help with that. | 19:29 |
| lifeless | jeblair: yes.what sdague's patch does thats unusual is make it the main targets that are constrainted, not the -constraints ones | 19:29 |
| lifeless | jeblair: this is different to what we wrote up for the project testing interface etc; but I see that social construct as fairly straight forward editable | 19:30 |
| *** sridhar_ram1 is now known as sridhar_ram | 19:30 | |
| *** unicell1 has joined #openstack-infra | 19:30 | |
| *** fawadkhaliq has quit IRC | 19:30 | |
| lifeless | https://review.openstack.org/#/c/266625/ and https://review.openstack.org/#/c/255656/ provide some context there | 19:30 |
| jeblair | lifeless, sdague: i think i agree that the mechanics of 267096 will work iff the standard tox env is run with the constraints zuul git prep | 19:31 |
| jeblair | lifeless, sdague: but that isn't the case now, is it? so that would need to happen first? | 19:31 |
| sdague | jeblair: yes | 19:31 |
| sdague | which is actually where I was trying to start the conversation | 19:32 |
| jeblair | and in order to do that, we should probably update the PTI as lifeless says? | 19:32 |
| sdague | will I get objections to making that change? | 19:32 |
| *** unicell has quit IRC | 19:33 | |
| sdague | well, actually, it would mean we wouldn't have to do those updates | 19:33 |
| sdague | because we'd be able to use the old names | 19:33 |
| sdague | which would also save use retraining the world | 19:33 |
| anteaya | retraining the world takes about 12 - 18 months and is undone if other changes requireing retraining occur during that period | 19:34 |
| anteaya | in my experience | 19:34 |
| mordred | anteaya: I'd like to retrain you on that | 19:34 |
| mordred | ;) | 19:34 |
| jeblair | sdague: side topic -- why don't we update the default envs in tox.ini? | 19:35 |
| lifeless | sdague: so the pti is already updated, the ones I linked to are updating the manual | 19:35 |
| anteaya | mordred: it's just me, so depending on your approach you might be done in less than 7 days | 19:35 |
| anteaya | mordred: if you can retrain the world in less than 12 months, I'd like to see how you do it | 19:35 |
| lifeless | sdague: https://governance.openstack.org/reference/cti/python_cti.html | 19:35 |
| jeblair | (i love that the name for the consistent testing interface is inconsistent) | 19:36 |
| lifeless | sdague: when we were doing the constraints work, we got steered by infra to making everything explicit in the project testing interface | 19:36 |
| sdague | jeblair: update them to what? | 19:37 |
| lifeless | sdague: so I'm assuming that keeping the docs and the expectations and so on all consistent is still important | 19:37 |
| jeblair | sdague: envlist = py34-constraints,py27-constraints,functional,pep8,pip-missing-reqs | 19:37 |
| sdague | jeblair: that assumes that people use tox without -e for running tests | 19:37 |
| sdague | which they rarely do, in my experience | 19:37 |
| jeblair | sdague: ok | 19:38 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Add cron job to run weekly prune the data set of db https://review.openstack.org/176564 | 19:38 |
| jeblair | sdague: i feel like ultimately, you are suggesting reversing the approach from a project opt-in to an opt-out? | 19:38 |
| dougwig | well, we trained them not to use -e, by insisting on including py3 env's before they worked. | 19:38 |
| jeblair | sdague: as projects that don't use constraints would need to update their jobs to use new "py27-without-constraints" macros, templates, etc.. | 19:38 |
| jeblair | dougwig: that was clever of us; considering the _only_ reason that's there is for developer convenience :) | 19:39 |
| jeblair | it certainly has nothing to do with ci :) | 19:39 |
| *** Keedya_ has joined #openstack-infra | 19:39 | |
| fungi | dougwig: meh, we also included py26 envs for the longest time even though most devs didn't have a 2.6 interpreter installed | 19:39 |
| sdague | jeblair: no, what I'm saying is that we have 1 developer interface for running tests | 19:39 |
| * jeblair thinks it should just be py27 | 19:39 | |
| sdague | tox -e py27 | 19:39 |
| openstackgerrit | Merged openstack-infra/project-config: Fix ENABLED_SERVICES concat failure for stable branch jobs https://review.openstack.org/267076 | 19:39 |
| sdague | the project decides what happens behind the scenes | 19:40 |
| sdague | it's a project testing interface, not project testing implementation | 19:40 |
| *** piet has quit IRC | 19:40 | |
| jeblair | sdague: except your proposal is predicated on running zuul-cloner beforehand | 19:40 |
| dansmith | +1 for one tox target | 19:40 |
| *** boris-42 has joined #openstack-infra | 19:40 | |
| jeblair | sdague: which means updating the standard env for running tox on all projects (openstack, stackforge, infra, etc) to clone the requirements repo | 19:40 |
| jeblair | sdague: any project that doesn't need/want that to happen would have to opt out | 19:41 |
| sdague | how big of a cost is that? | 19:41 |
| * jeblair checks if he's still allowed to say 'stackforge' | 19:41 | |
| jeblair | (yes, it hasn't merged yet) | 19:41 |
| sdague | so, I guess I don't understand | 19:41 |
| sdague | because we're talking about zuul cloner, in upstream ci, making another repo available on disk | 19:41 |
| sdague | far away from developers laptop | 19:41 |
| sdague | that extra repo on disk doesn't hurt anyone that doesn't want to use it | 19:42 |
| *** tsg has joined #openstack-infra | 19:42 | |
| jeblair | sdague: it will generate some extra load and add time to every job, regardless of whether they use it | 19:43 |
| jeblair | sdague: i do not know how much, but i ask that you consider that | 19:43 |
| *** trown|lunch is now known as trown | 19:43 | |
| sdague | compared to the 40 minutes to spin the node? | 19:44 |
| *** rkukura has quit IRC | 19:44 | |
| lifeless | afk sorry, -> child prep | 19:44 |
| sdague | this seems really minor | 19:44 |
| AJaeger | do we need an extra repo - or just a single file from it that we could just download or distribute with puppet ? | 19:45 |
| jeblair | sdague: it's an extra cost to *every* tox job. i do not know the cost. would you be willing to consider that in your proposal? | 19:45 |
| *** darvon has quit IRC | 19:46 | |
| *** piet has joined #openstack-infra | 19:46 | |
| sdague | I'm not sure what I'm considering honestly, so I don't know | 19:46 |
| jeblair | sdague: okay, well we should keep it in mind | 19:47 |
| *** rkukura has joined #openstack-infra | 19:47 | |
| jeblair | sdague: so to your question about objections -- the only ones i can think of are: 1) we should agree on what we want to do globally and update the PTI as necessary to reflect the consensus and define the standard | 19:47 |
| openstackgerrit | Merged openstack-infra/system-config: Add ci-backup-rs-ord.openstack.org to cacti https://review.openstack.org/267050 | 19:48 |
| sdague | jeblair: ok, I guess I don't see where in the PTI this level of implementation is specified | 19:48 |
| jeblair | sdague: "Projects desiring to support constrained unit tests must support: tox -epy27-constraints, tox -epy34-constraints" | 19:49 |
| jeblair | sdague: 2) as i understand it, we would need to zuul-clone the reqs repo before running 'tox -epy27' in zuul -- that has an unknown cost to job runtime and infrastructure (git server / zuul merger) load, that should be evaluated. if we decide that we don't want to bear that cost for all tox jobs, we would need to invert the logic around running tox jobs to allow projects to opt-out of that. | 19:50 |
| sdague | is that in the repo? | 19:50 |
| sdague | git grep is failing me | 19:50 |
| jeblair | sdague: it's at https://governance.openstack.org/reference/cti/python_cti.html | 19:50 |
| sdague | ah ok | 19:50 |
| *** ayoung has quit IRC | 19:51 | |
| *** ldnunes_ has joined #openstack-infra | 19:51 | |
| jeblair | sdague: those are the only objections i can come up with. and to be clear, i think they are things we can work through and overcome. i'm trying to be helpful by thinking of the negative things you asked for. :) | 19:51 |
| sdague | jeblair: ok, sorry, I guess it was more rhetorical question... :) | 19:52 |
| jeblair | bwahaha | 19:52 |
| *** mriedem_meeting is now known as mriedem | 19:52 | |
| sdague | if I push this patch, am I in for a -1 fest that means I should give up | 19:52 |
| *** ldnunes has quit IRC | 19:53 | |
| jeblair | sdague: i think it's not trivial -- i think this touches the project testing interface which we keep standard because our heads explode if things behave differently (in certain ways -- particularly this one where we're talking about dependencies) between projects. | 19:53 |
| sdague | it is unit tests, when do we do dependenies between projects in unit tests? | 19:55 |
| *** darvon has joined #openstack-infra | 19:55 | |
| *** mpavone has joined #openstack-infra | 19:55 | |
| jeblair | sdague: there was a close-paren there. | 19:56 |
| *** mudassirlatif has quit IRC | 19:56 | |
| jeblair | sdague: the 'between projects' was 'behaving differently'. | 19:56 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: always make upper-constraints available to python jobs https://review.openstack.org/267133 | 19:56 |
| sdague | ah, ok | 19:56 |
| *** aopincar_ has joined #openstack-infra | 19:56 | |
| jeblair | adding words made less clear | 19:56 |
| *** davidlenwell has quit IRC | 19:57 | |
| sdague | well, honestly I think there is too much belief that things aren't already different between projects :) | 19:57 |
| sdague | I don't see this as substantially making that any worse | 19:57 |
| sdague | and it hugely helps in reducing confusion in getting folks over | 19:57 |
| jeblair | sdague: i'm not saying it's a bad idea | 19:57 |
| sdague | so I think https://review.openstack.org/267133 is what is needed | 19:57 |
| *** sridhar_ram has quit IRC | 19:58 | |
| *** davidlenwell has joined #openstack-infra | 19:58 | |
| mordred | sdague: we'd like to in-general trend towards the Platonic idea rather than away from, even if the shadows and the objects are not the same today | 19:58 |
| mordred | sdague: but I'm just drive-by commenting at the moment, so ignore me | 19:58 |
| sdague | mordred: sure, that's fine, but I feel like that sword gets wielded a lot in the abstract | 19:59 |
| mordred | (tried to figure out a way to get assymptotically into that sentence but failed) | 19:59 |
| mordred | sdague: indeed | 19:59 |
| *** mpavone has quit IRC | 19:59 | |
| sdague | and it usually disregards all the actually important differences that come with separate team cultures | 19:59 |
| mordred | sdague: in a non-drive y comment way - I think that th abstract is very important to keep track of, and balancing the practicality of the moment with the goal of the abstract is the hard part of the job | 19:59 |
| *** baoli_ has quit IRC | 20:00 | |
| mordred | sdague: see, we were so close to being on the same page - I largely see the "important differences that come with separate team cultures" as a big bug in OpenStack. nova having its own culture is a problem, not a feature | 20:00 |
| mordred | sdague: and projects having that own culture is one of the thing that leads to cliques and boys-clubs and intra-project distrust | 20:01 |
| jeblair | sdague: i hope you are not dismissing my concerns as abstract -- i am VERY seriously concerned about the complexity of the constraints system which i believe no one really fully understands in its entirety, and i do not think we should lightly take the decision to ignore the cross-project standard we have established around it | 20:01 |
| mordred | however much it may be reality that it exists | 20:01 |
| jeblair | sdague: i repeat *we can change that standard* | 20:01 |
| *** tonytan4ever has quit IRC | 20:03 | |
| *** bharathm has quit IRC | 20:04 | |
| *** eharney has quit IRC | 20:06 | |
| SamYaple | mordred: For whats its worth I agree with you on the "own culture" thing. I have been trying to get Kolla to a more generic way of doing things where possible | 20:06 |
| *** rguillebert has quit IRC | 20:07 | |
| *** ddieterly has quit IRC | 20:09 | |
| openstackgerrit | Merged openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder https://review.openstack.org/263349 | 20:09 |
| *** claudiub has quit IRC | 20:10 | |
| *** esikachev has joined #openstack-infra | 20:11 | |
| *** rkukura has quit IRC | 20:14 | |
| *** ldnunes_ has quit IRC | 20:14 | |
| *** rajinir has quit IRC | 20:15 | |
| jeblair | proposed at least as a discussion starting point: https://review.openstack.org/267140 Change the default set of tox jobs to run | 20:15 |
| pc_m | Hi infra folks! I need some guidance on how to handle getting neutron to use the neutron-lib project that is being developed. | 20:16 |
| *** annegentle has quit IRC | 20:17 | |
| pc_m | dougwig created a commit, https://review.openstack.org/#/c/242219, that uses the neutron-lib. It fails several dsvm based tests. | 20:17 |
| *** terriyu has joined #openstack-infra | 20:17 | |
| *** annegentle has joined #openstack-infra | 20:17 | |
| *** eliqiao has quit IRC | 20:17 | |
| pc_m | It *looks* like we could add neutron-lib to the list of PROJECTS in devstack-vm-gate-wrap.sh. Would that be how to approach this? | 20:17 |
| dhellmann | fungi : https://review.openstack.org/267143 move extra-atcs into projects.yaml | 20:18 |
| *** dims has quit IRC | 20:18 | |
| fungi | dhellmann: thanks a bunch! | 20:18 |
| *** esikachev has quit IRC | 20:18 | |
| *** dims has joined #openstack-infra | 20:18 | |
| dhellmann | fungi : I'm pretty sure there are other tools that need to be updated, but that should get us started | 20:18 |
| pc_m | Would that work, or is there some other way to get neutron-lib installed so that neutron can reference it? | 20:19 |
| pc_m | There are 6 voting tests that fail. | 20:20 |
| *** eliqiao has joined #openstack-infra | 20:20 | |
| fungi | dhellmann: structure data ftw. also would be nice to have the expires-in be something easier to match on, but i guess that's mainly just helpful for the tc to know when to expunge them from the file | 20:20 |
| *** rcernin has quit IRC | 20:20 | |
| piet | Sorry to randomize, but looking forward to meeting some of you at the mid-cycle in Fort Collins! | 20:20 |
| *** lucasagomes is now known as lucas-dinner | 20:20 | |
| anteaya | piet: what midcycle? | 20:21 |
| dhellmann | fungi : we can adjust that to be a date format we can parse or something. or just use all numbers? let's do that as a follow-on based on what the tool needs | 20:21 |
| anteaya | piet: the infra mid-cycle? | 20:21 |
| fungi | piet: you're coming to work on the infra-cloud deployment? | 20:21 |
| *** Keedya_ has quit IRC | 20:21 | |
| piet | fungi: yeah.......no. | 20:21 |
| anteaya | pc_m: um, have you chatted with dougwig about this? | 20:22 |
| fungi | dhellmann: yeah, i don't actually have a use for that piece of metadata at the moment anyway | 20:22 |
| dhellmann | fungi : ok | 20:22 |
| anteaya | piet: do you mean dinner when we are in fort collins? | 20:22 |
| piet | Yeah | 20:22 |
| pc_m | anteaya: sure. He asked if I could look into the faillures. | 20:22 |
| anteaya | oh okay yeah, well be pingable on irc and I'll do my best to relay dinner plans so you can join in where appropriate | 20:23 |
| piet | anteaya: This sprint - https://wiki.openstack.org/wiki/Sprints/InfraMitakaSprint | 20:23 |
| anteaya | piet: I'm looking forward to seeing you too | 20:23 |
| anteaya | piet: right | 20:23 |
| dougwig | pc_m: this is not the appropriate place for this. | 20:23 |
| dougwig | pc_m: let's go off channel. | 20:23 |
| anteaya | piet: also is ux having a sprint? | 20:23 |
| anteaya | piet: Zara mentioned a ux sprint, was that your group? | 20:23 |
| piet | anteaya; Yeah, one in London (assuming I get approval) that coincides with the Operators Summit in Manchester | 20:24 |
| *** _nadya_ has quit IRC | 20:24 | |
| lifeless | sdague: jeblair: I'm back. Where did the discussion get to? | 20:24 |
| piet | anteaya: One in Portland that coincide with the Horizon sprint | 20:24 |
| anteaya | piet: okay can you pencil in anything on the sprints wikipage? | 20:25 |
| piet | sure | 20:25 |
| anteaya | piet: Zara mentioned something and I had nothing to reference on the sprints page | 20:25 |
| anteaya | piet: thank you | 20:25 |
| piet | Give me a day to confirm travel approval | 20:25 |
| *** e0ne has quit IRC | 20:25 | |
| anteaya | piet: understood | 20:26 |
| *** rkukura has joined #openstack-infra | 20:26 | |
| *** Sukhdev has quit IRC | 20:26 | |
| fungi | pleia2: do you have any pointers for how we can collect contributor details from zanata yet? | 20:26 |
| pleia2 | fungi: I think Daisy was working directly with the Zanata folks on that, I don't have any updates | 20:27 |
| gtmanfred | can I get someone to look at this? https://review.openstack.org/#/c/266532/ | 20:27 |
| *** ericksonsantos has quit IRC | 20:27 | |
| fungi | pleia2: okay, just hoping in can integrate something into this new tool i'm writing so that it doesn't end up being a manual process | 20:27 |
| pleia2 | fungi: unfortunately at the summit all I learned is that they're overhauling parts of their API to better support it (and removing some old ooptions) | 20:28 |
| sdague | lifeless: I don't know, I've proposed a pti change though - https://review.openstack.org/#/c/267149/1/reference/cti/python_cti.rst | 20:28 |
| pleia2 | so now is probably a bad time | 20:28 |
| fungi | pleia2: oh, that's unfortunate. i don't suppose there's much we can parse out of its git backend either? | 20:28 |
| pleia2 | fungi: nope | 20:29 |
| clarkb | there is a mysql db ya? | 20:29 |
| clarkb | maybe we can parse out of that? | 20:29 |
| pleia2 | there is, but it's a bit... complicated | 20:29 |
| pleia2 | this table, see that table for association | 20:29 |
| anteaya | :( | 20:29 |
| fungi | i'm okay with just waiting for an api in that case (hopefully one that doesn't require admin privs) | 20:29 |
| pleia2 | we specifically asked that whatever it is not require admin | 20:29 |
| fungi | excellent | 20:30 |
| pleia2 | they were pushing for requiring it, since some projects aren't open source everyone-sees-everything like us | 20:30 |
| pleia2 | but I think we won an option in the admin interface to allow for it | 20:30 |
| *** jaosorior has quit IRC | 20:30 | |
| fungi | i'm just about at feature parity with our old atcs script, plus some nice-to-haves like duplicate address detection and extra-atc merging | 20:30 |
| *** eharney has joined #openstack-infra | 20:30 | |
| *** jaosorior has joined #openstack-infra | 20:30 | |
| *** bharathm has joined #openstack-infra | 20:31 | |
| pleia2 | it's about time for me to drop an email to Daisy about it anyway | 20:31 |
| anteaya | fungi: yay | 20:31 |
| pleia2 | I think she actually has something to poll for ATC for i18n | 20:31 |
| anteaya | and yay for dup detection | 20:31 |
| pleia2 | but it could have been something manual | 20:31 |
| *** esikachev has joined #openstack-infra | 20:32 | |
| pleia2 | lunchtime | 20:32 |
| *** jsavak has quit IRC | 20:33 | |
| anteaya | pleia2: enjoy lunch | 20:33 |
| *** mudassirlatif has joined #openstack-infra | 20:33 | |
| *** otter768 has joined #openstack-infra | 20:34 | |
| *** jaosorior has quit IRC | 20:34 | |
| *** tonytan4ever has joined #openstack-infra | 20:35 | |
| *** ddieterly has joined #openstack-infra | 20:35 | |
| *** jsavak has joined #openstack-infra | 20:35 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: always make upper-constraints available to python jobs https://review.openstack.org/267133 | 20:35 |
| sdague | jeblair: updated with Depends-On for governance change | 20:36 |
| anteaya | mordred: so you are aware that currently nodepool nodes on bluebox are 100% error, yeah? | 20:36 |
| mordred | anteaya: I am not | 20:37 |
| mordred | jlk: ^^ your cloud is broken | 20:37 |
| *** rguillebert has joined #openstack-infra | 20:37 | |
| anteaya | thanks | 20:37 |
| mordred | yup | 20:37 |
| *** daemontool has quit IRC | 20:37 | |
| mordred | that sure does look like error | 20:37 |
| anteaya | big box of red | 20:37 |
| jeblair | er wait | 20:37 |
| * anteaya waits | 20:38 | |
| jeblair | we probably just haven't restarted after changing the flavor | 20:38 |
| anteaya | ah ha | 20:38 |
| mordred | oh! yes | 20:38 |
| mordred | that, in fact, would likely do it | 20:38 |
| anteaya | jlk: might be our end | 20:38 |
| *** daemontool has joined #openstack-infra | 20:38 | |
| jlk | whew! | 20:38 |
| *** otter768 has quit IRC | 20:38 | |
| * mordred hands jlk a nice cold towel | 20:38 | |
| *** markmcclain has joined #openstack-infra | 20:38 | |
| jeblair | though, we were in the middle of a conversation with jlk about storage and we were waiting on wrapping that up to see if we were going to change the flavor again :) | 20:38 |
| jlk | oh right. | 20:39 |
| jlk | so... | 20:39 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit https://review.openstack.org/267066 | 20:39 |
| * mordred takes the towel back and uses it to chastise jlk | 20:39 | |
| jlk | that's. That's an interesting story | 20:39 |
| * jeblair settles in for storytime | 20:39 | |
| * anteaya pours hot chocolate | 20:39 | |
| *** hashar has joined #openstack-infra | 20:39 | |
| jlk | this is something that shouldn't be spread around | 20:40 |
| jlk | but | 20:40 |
| jlk | your cloud is slightly misconfigured. | 20:40 |
| jeblair | i won't tell anyone | 20:40 |
| jlk | The machines we get from SoftLayer have two raid arrays. One array that is a mirror of "slow" disks for the OS. We asked for 500G per the BOM, we got 1TB. | 20:40 |
| jlk | There is a second array, a 1.2TB array of 4 fast disks. | 20:40 |
| * rockyg ,s ears perk up | 20:40 | |
| jlk | 10K RPM SAS | 20:40 |
| *** ddieterly has quit IRC | 20:41 | |
| jlk | At the time your cluster was deployed, the code to deploy it hadn't yet learned how to deal with two disk sets | 20:41 |
| jlk | prior to using SoftLayer we just had one big fast array | 20:41 |
| jlk | so the code treated this system the same way, so all your instances are landing on that 2 disk slow mirror | 20:41 |
| jlk | and the 4 disk raid 10 mirror is sitting unused. | 20:41 |
| jlk | Capacity wise, they nearly matched up, looked right | 20:42 |
| * anteaya waits for the part where bluebox gets faster | 20:42 | |
| jlk | We're working on the orchestration to migrate instance data from the slow mirror to the fast raid 10 | 20:42 |
| jlk | and going to schedule the maint window to execute it | 20:42 |
| jeblair | anteaya: i think it's coming up! | 20:42 |
| jlk | We're going to put y'all on the top of the list to execute | 20:42 |
| mordred | yay! | 20:42 |
| mordred | we like this | 20:42 |
| anteaya | yay! | 20:42 |
| anteaya | woooooo | 20:42 |
| anteaya | jlk: thank you | 20:43 |
| jlk | What would help at that time, is if ya'll could purge your cloud | 20:43 |
| jlk | so we don't hav eto rsync over any instance data | 20:43 |
| *** apoorvad has quit IRC | 20:43 | |
| jeblair | jlk: so at the end of this, we will still have ~1TB per hypervisor to share among root and ephemeral volumes? | 20:43 |
| jlk | if there is ever a good time for you to do that, we could coordinate with you and get it done. | 20:43 |
| anteaya | jlk: so purging the cloud is just setting bluebox to 0 max_servers in nodepool.yaml? | 20:43 |
| jeblair | jlk: just a different 1TB? | 20:43 |
| anteaya | sorry that was for jeblair | 20:43 |
| anteaya | jeblair: ^^ | 20:43 |
| jlk | jeblair: correct. It's about 1.2TB usable. | 20:43 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Prepare neutron-lib for pypi releases https://review.openstack.org/267156 | 20:44 |
| jeblair | jlk, anteaya: it's easy for us to stop using the instances; but it's a moderate amount of work for us to rebuild the mirror (maybe ~1 or 2 days)... how hard would it be to have 1 instance with 300G of data to move? | 20:45 |
| anteaya | ah, the mirror | 20:45 |
| jlk | jeblair: not hard, just makes the outage longer. We'll have to stop the instance and make sure all the file handlers are freed | 20:45 |
| jlk | I'm hoping we can do rsync's on the live data leading up to the outage window, so it'd be a quick sync up for any last minute changes | 20:46 |
| rm_work | this issue is nothing to do with our tests and a recheck should resolve, right? http://logs.openstack.org/31/267031/1/check/gate-neutron-lbaasv2-dsvm-listener/52c51d0/console.html#_2016-01-13_19_56_12_740 | 20:46 |
| rm_work | ^^ just happened | 20:46 |
| rm_work | well, semi-recently | 20:46 |
| jeblair | jlk: okay, that's probably best then if you can accomodate it; if we later find it's too much trouble, we can rebuild it | 20:46 |
| jlk | okay. | 20:47 |
| *** dimtruck is now known as zz_dimtruck | 20:47 | |
| *** sridhar_ram has joined #openstack-infra | 20:47 | |
| jlk | big apologies on this one. Our 2.0 product is all set to handle the disk separation. | 20:47 |
| jeblair | no worries! we like helping people find problems! | 20:47 |
| jeblair | and even solutions | 20:48 |
| jeblair | occasionally | 20:48 |
| jeblair | jlk, clarkb, mordred, anteaya: sounds like no changes for the moment so i'll go ahead and restart nodepool to pick up the new flavor now | 20:48 |
| anteaya | k | 20:48 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: REduce bluebox max_servers from 39 to 0 https://review.openstack.org/267160 | 20:49 |
| anteaya | there's the patch for when we want to merge it | 20:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: refactor list_deliverable_changes.py prior to move to project-config https://review.openstack.org/267161 | 20:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: refactor script for commenting on launchpad bugs prior to move to project-config https://review.openstack.org/267162 | 20:49 |
| *** rcernin has joined #openstack-infra | 20:49 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Reduce bluebox max_servers from 39 to 0 https://review.openstack.org/267160 | 20:50 |
| jeblair | gertty needs a history screen -- like "these are the last X changes you looked at" | 20:51 |
| anteaya | jlk: do you want to -1 on 267160 for now and +1 when you know when the maintenance window is scheduled? | 20:51 |
| dhellmann | fungi, jeblair : I would appreciate your thoughts on those 2 patches in release-tools ^^ as precursors to moving the scripts over to project-config | 20:51 |
| jlk | anteaya: sure can! | 20:52 |
| anteaya | jlk: thank you, helps us get the timing right | 20:52 |
| jlk | I really can't get used to the new gerrit UI | 20:53 |
| anteaya | jlk: you are in good company | 20:53 |
| jeblair | jlk: how do you feel about text-mode console applications? | 20:54 |
| jlk | Well, I already use "hub" to manage github PRs, so... favorable. I know there are tools out there for gerrit, it's just not party of my day to day work life so I haven't bothered yet. | 20:54 |
| jeblair | jlk: well, if it every becomes a significant part of your work, i wrote gertty which is like 'mutt' for gerrit: https://pypi.python.org/pypi/gertty/1.3.1 | 20:55 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: default to sending mail directly via lists.openstack.org https://review.openstack.org/267166 | 20:55 |
| jlk | mutt.... ouch. | 20:55 |
| jlk | :D | 20:55 |
| jeblair | jlk: otherwise, they are working on an actual modern javascript interface called polygerrit, but it's a ways away | 20:56 |
| jeblair | (it's actually more like gnus for gerrit, but more people use mutt so i say that) | 20:56 |
| *** armax has quit IRC | 20:56 | |
| mordred | jlk: gertty is teh ossum | 20:56 |
| jlk | just as long as it isn't full of ncurses | 20:56 |
| jlk | or emacs key bindings | 20:57 |
| jeblair | jlk: oh no. no ncurses. direct ansi codes. emacs keybindings are default, vi are optional. | 20:58 |
| jhesketh | Morning | 20:59 |
| *** baoli has joined #openstack-infra | 20:59 | |
| anteaya | morning jhesketh | 20:59 |
| jroll | whoa, vi bindings exist in gertty now? | 21:00 |
| * jroll applies config | 21:00 | |
| jeblair | wow, i'm watching the story of a node coming online by seeing spikes on the api graphs appear in grafana | 21:00 |
| mordred | neat! | 21:01 |
| *** ari has quit IRC | 21:01 | |
| mordred | jeblair: which daskboard? | 21:01 |
| jeblair | jroll: it has support for multi-key commands now, so you can ':q', etc... there's some examples of how to do that in the config -- if someone who actually knows vi writes a set of bindings we can merge it in to gertty so it's built-in | 21:01 |
| jeblair | mordred: http://grafana.openstack.org/dashboard/db/nodepool-blue-box | 21:01 |
| *** ekarlso has quit IRC | 21:01 | |
| *** ekarlso has joined #openstack-infra | 21:01 | |
| jroll | jeblair: ah, I see :) | 21:02 |
| *** gildub has joined #openstack-infra | 21:03 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/project-config: Identity API v3 only non-voting job in Tempest https://review.openstack.org/267171 | 21:03 |
| *** rockyg has quit IRC | 21:03 | |
| *** rcernin has quit IRC | 21:04 | |
| samueldmq | sdague: mtreinish ^ change for tempest | 21:04 |
| samueldmq | mtreinish: that was quick! thanks | 21:05 |
| *** Keedya_ has joined #openstack-infra | 21:06 | |
| *** rockyg has joined #openstack-infra | 21:07 | |
| *** Keedya_ has quit IRC | 21:07 | |
| *** akshai has quit IRC | 21:08 | |
| *** zz_dimtruck is now known as dimtruck | 21:08 | |
| anteaya | samueldmq: great commit, message | 21:08 |
| anteaya | thank you | 21:08 |
| anteaya | sorry for the errand comma in there | 21:08 |
| *** ddieterly has joined #openstack-infra | 21:08 | |
| anteaya | s/errand/errant | 21:09 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add a vi keymap https://review.openstack.org/267175 | 21:09 |
| samueldmq | anteaya: thanks, I tried to make it very clear what the state is and what it is about | 21:09 |
| *** baoli has quit IRC | 21:09 | |
| jeblair | jroll: here i made it easy in https://review.openstack.org/267175 :) | 21:09 |
| samueldmq | anteaya: thanks for your feedback :) | 21:10 |
| *** baoli has joined #openstack-infra | 21:10 | |
| anteaya | samueldmq: you were very successful, what you gave me is what I hope for in all commit messages, please keep doing that | 21:10 |
| jeblair | jroll: i only know how to quit vi, so that's all i can do for now | 21:11 |
| samueldmq | anteaya: I will try with my best, thanks :) | 21:11 |
| anteaya | samueldmq: :) | 21:11 |
| anteaya | jeblair: ha ha ha | 21:11 |
| *** ayoung has joined #openstack-infra | 21:11 | |
| *** yonglihe has quit IRC | 21:11 | |
| jroll | jeblair: ha! nice | 21:12 |
| jroll | I'll see what I can do | 21:12 |
| anteaya | when I first opened vi, at home, far from user groups, fresh to linux from windows and no connection to the internet, I opened vi and couldn't close it | 21:12 |
| *** akshai has joined #openstack-infra | 21:12 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-pip: Added extra-index-url to pip.conf https://review.openstack.org/267107 | 21:12 |
| anteaya | I've had car accidents that caused me less stress | 21:12 |
| mtreinish | jeblair: you're better than me I don't know how to exit emacs besides 'killall -9 emacs' | 21:12 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add a vi keymap https://review.openstack.org/267175 | 21:13 |
| *** amrith is now known as _amrith_ | 21:13 | |
| jeblair | mtreinish: no that's right. | 21:13 |
| *** esikachev has quit IRC | 21:13 | |
| dougwig | the modal editor that will not die. | 21:13 |
| *** ig0r_ has quit IRC | 21:14 | |
| jeblair | anteaya: if you've never read http://www.gnu.org/fun/jokes/ed-msg.html you may enjoy it :) | 21:15 |
| jeblair | vi and emacs have nothing on ed | 21:15 |
| anteaya | I've used ed | 21:15 |
| anteaya | I like it | 21:16 |
| anteaya | ha ha ha | 21:16 |
| anteaya | oh but that was intentionally | 21:16 |
| anteaya | there was one time I said open the file and it opened in ed, boy that was another fun one | 21:16 |
| anteaya | some of my most terrifying life memories are text editor related | 21:17 |
| jeblair | i didn't think anyone had ever used ed intentionally | 21:17 |
| anteaya | I read a tutorial | 21:17 |
| anteaya | and though meh, I'll try it, so I did | 21:17 |
| dougwig | i've used it when /bin was the only thing not hosed. but not by choice. | 21:18 |
| *** akshai has quit IRC | 21:19 | |
| *** baoli has quit IRC | 21:21 | |
| *** baoli has joined #openstack-infra | 21:22 | |
| anteaya | I can feel your neck tendons creak | 21:22 |
| anteaya | jeblair: thanks that was fun | 21:22 |
| *** kgiusti has quit IRC | 21:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:23 | |
| *** ig0r_ has joined #openstack-infra | 21:23 | |
| *** sdake has quit IRC | 21:26 | |
| *** angdraug has quit IRC | 21:28 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 21:30 |
| *** armax has joined #openstack-infra | 21:30 | |
| mordred | jeblair: I want you to know I just fell into a hole of reading the texinfo manual for ed | 21:31 |
| *** ig0r_ has quit IRC | 21:31 | |
| *** aysyd has quit IRC | 21:31 | |
| gtmanfred | can I get someone to look at this patch for shade? https://review.openstack.org/#/c/266532/ | 21:32 |
| *** jtomasek has quit IRC | 21:32 | |
| anteaya | jeblair: nicely done | 21:33 |
| *** baoli has quit IRC | 21:33 | |
| *** baoli has joined #openstack-infra | 21:34 | |
| *** salv-orlando has quit IRC | 21:34 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit https://review.openstack.org/267066 | 21:34 |
| *** salv-orlando has joined #openstack-infra | 21:35 | |
| *** apoorvad has joined #openstack-infra | 21:35 | |
| anteaya | fungi: there is a gerrit bug up for the needed to parse change successfully merged for the timestamp of the merge if I remember correctly, could that be added to the commit message for your script as well? | 21:36 |
| Shrews | gtmanfred: Could you educate me on exactly why this is needed? TBH, I'm quite concerned about the security implications of storing plain text passwords in possibly cached data. | 21:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors https://review.openstack.org/253521 | 21:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 21:36 |
| * anteaya followed the link to the bug for viewing emails via REST and reviewed it in gerrit's gerrit | 21:37 | |
| gtmanfred | Shrews: for when servers are built, it is the initial password. I didn't think it cached the data from a build, but if you aren't passing a key for the initial build, I use the initial passwod, and then use my saltstack automation to put my ssh keys on the server and disable password authentication | 21:38 |
| fungi | anteaya: it's actually mentioned in at least a couple code comments in the script itself with todo tasks called out | 21:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh https://review.openstack.org/267117 | 21:39 |
| anteaya | fungi: oh sorry, was skimming, I will look closer | 21:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Bump APScheduler to >=3.0 https://review.openstack.org/262017 | 21:40 |
| fungi | anteaya: that way it survives a bit better than getting hidden away in a commit message | 21:40 |
| *** ajmiller has quit IRC | 21:40 | |
| anteaya | so it is | 21:40 |
| anteaya | yup, your way is better | 21:41 |
| anteaya | clearly spelled out in line 111 for example | 21:41 |
| fungi | anteaya: it will basically allow us to gut that really hacky function and just check a value in the change data we're already querying instead | 21:41 |
| Shrews | gtmanfred: i'm still not clear as to why the server object must keep the initial password around AFTER the server has been built? | 21:41 |
| greghaynes | ianw: just kicked off a recheck on the branch pruning patch - should be able to pass now | 21:42 |
| *** rfolco has quit IRC | 21:42 | |
| gtmanfred | because it is never returned, so if it is auto assigned by the provider, you never get it back | 21:42 |
| anteaya | zaro: oh look this patch of yours has some helpful reviews! https://gerrit-review.googlesource.com/#/c/73651/ | 21:42 |
| fungi | anteaya: and it will also let us cut down on some additional query bloat since we no longer need to get gerrit to include comment details on each change | 21:42 |
| gtmanfred | rackspaces cloud auto assigns a password, and only the initial return of the server object has that admin_pass in it | 21:42 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Make zaqarclient-dsvm-functional job as voting https://review.openstack.org/266596 | 21:42 |
| anteaya | fungi: yup, it will be much cleaner after 73651 merges and is backported | 21:43 |
| anteaya | fungi: thanks sorry for being slow, yay script | 21:43 |
| loquacities | anteaya, jeblair: you rang? | 21:43 |
| gtmanfred | Shrews: if it was only for passing admin_pass to the object, it wouldn't be needed, but for one that is assigned, there is no way to get the password after that first return | 21:43 |
| anteaya | loquacities: hello | 21:43 |
| loquacities | hi! | 21:44 |
| *** ajmiller has joined #openstack-infra | 21:44 | |
| anteaya | loquacities: just wanted to get your on the record response to 259569 so I can review | 21:44 |
| *** ig0r_ has joined #openstack-infra | 21:44 | |
| anteaya | loquacities: sdague feels you have reached agreement | 21:44 |
| *** akshai has joined #openstack-infra | 21:44 | |
| Shrews | gtmanfred: ah, i think i'm starting to understand | 21:44 |
| gtmanfred | cool | 21:45 |
| *** ig0r_ has quit IRC | 21:45 | |
| loquacities | agreement insofar as 259569 can go ahead, i guess | 21:45 |
| *** baoli has quit IRC | 21:45 | |
| anteaya | loquacities: I can accept that | 21:45 |
| anteaya | thank you | 21:45 |
| loquacities | np | 21:45 |
| *** baoli has joined #openstack-infra | 21:45 | |
| Shrews | gtmanfred: to be honest, i'm not entirely sure this is something we'd want to support. mordred, do you have thoughts on this? | 21:46 |
| mordred | hrm | 21:46 |
| openstackgerrit | gordon chung proposed openstack/requirements: bump ceilometerclient to 2.2.1 https://review.openstack.org/266440 | 21:46 |
| mordred | I mean, it _is_ a part of the nova api thing - but I certainly personally have no interest in admin passwords | 21:47 |
| mordred | it doesn't bother me supporting it though | 21:47 |
| mordred | Shrews: what's the issue? | 21:47 |
| gtmanfred | https://review.openstack.org/#/c/266532/ | 21:47 |
| Shrews | mordred: my concern would be security implications, especially if someone is using caching | 21:47 |
| mordred | it seems to be mainly just passthrough, no? oh - I see what you're saying | 21:48 |
| anteaya | thanks loquacities | 21:48 |
| loquacities | np | 21:48 |
| Shrews | mordred: we could totally support it, but have the caveat that "hey, you could open yourself up for bad things" | 21:48 |
| mordred | it's because we're grabbing it from the initial return and then injecting it after the wait loop | 21:48 |
| *** piet has quit IRC | 21:49 | |
| *** Sukhdev has joined #openstack-infra | 21:49 | |
| *** dimtruck is now known as zz_dimtruck | 21:49 | |
| Shrews | mordred: if someone chooses file-backed dogpile backend caching, for example, they suddenly have clear text passwords in the cache file | 21:49 |
| Shrews | if i'm being overly paranoid, then the change is fine. just want us to be aware of the implications | 21:50 |
| mordred | yah | 21:50 |
| mordred | well | 21:50 |
| mordred | hrm | 21:50 |
| mordred | we could also put a filter in the cache-if for that ... | 21:51 |
| *** aopincar_ has quit IRC | 21:51 | |
| mordred | if the server has a password, don't cache it?... OH - wait, that's not going to get cached | 21:51 |
| mordred | this is the return value of the create call | 21:51 |
| *** dprince has quit IRC | 21:51 | |
| gtmanfred | yeah | 21:51 |
| mordred | we cache list-servers | 21:51 |
| gtmanfred | that was what I was thinking | 21:51 |
| *** sdake has joined #openstack-infra | 21:53 | |
| Shrews | mordred: get_server() will return the server FROM the cache (once it appears). then it gets modified. doesn't that change go back into the cache? | 21:54 |
| mordred | Shrews: not to my knowledge, no | 21:55 |
| *** rcernin has joined #openstack-infra | 21:55 | |
| mordred | Shrews: MAYBE using the in-memory cache option, but that is a memory leak anyway and is not suggested | 21:55 |
| mordred | but if it's a remote cache, like dbm or memcached, it'll cache-then-return and the objects after that point are not active-tied back to the cache | 21:56 |
| Shrews | mordred: if that's the case (my knowledge of dogpile is limited), then my worry is for naught | 21:56 |
| *** sdake_ has joined #openstack-infra | 21:56 | |
| *** hashar has quit IRC | 21:56 | |
| *** baoli has quit IRC | 21:57 | |
| mordred | Shrews: it's good to worry | 21:57 |
| *** sdake has quit IRC | 21:57 | |
| *** baoli has joined #openstack-infra | 21:58 | |
| Shrews | gtmanfred: in that case, please add a unit test that makes sure the password returned from initial create survives (not everyone can run the functional tests, but everyone should be able to run unit tests) | 22:00 |
| Shrews | basically an equivalent of your functional test | 22:00 |
| *** vgridnev has quit IRC | 22:00 | |
| gtmanfred | kk | 22:01 |
| *** ZZelle_ has joined #openstack-infra | 22:01 | |
| *** piet has joined #openstack-infra | 22:05 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Update constraint for os-client-config https://review.openstack.org/267195 | 22:08 |
| *** salv-orl_ has joined #openstack-infra | 22:09 | |
| *** baoli has quit IRC | 22:09 | |
| *** baoli has joined #openstack-infra | 22:10 | |
| *** burgerk has quit IRC | 22:10 | |
| *** zz_dimtruck is now known as dimtruck | 22:10 | |
| ianw | greghaynes: awesome, thanks, will keep an eye on it this time :) | 22:10 |
| *** rcernin has quit IRC | 22:10 | |
| *** salv-orlando has quit IRC | 22:12 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: Save the adminPass if returned on server create https://review.openstack.org/266532 | 22:13 |
| gtmanfred | Shrews: done, just waiting for tests to run now :) | 22:13 |
| *** jcoufal has quit IRC | 22:14 | |
| *** yamahata has quit IRC | 22:14 | |
| *** ddieterly has quit IRC | 22:16 | |
| *** jsavak has quit IRC | 22:18 | |
| openstackgerrit | Randall Burt proposed openstack-infra/project-config: Fix more syntax errors in gate-heat-templates-dsvm https://review.openstack.org/267201 | 22:18 |
| *** rlandy has quit IRC | 22:18 | |
| *** dimtruck is now known as zz_dimtruck | 22:20 | |
| *** smckee has joined #openstack-infra | 22:21 | |
| anteaya | smckee: hello | 22:21 |
| *** baoli_ has joined #openstack-infra | 22:21 | |
| ianw | http://xkcd.com/1629/ ... there really is a xkcd for everything | 22:21 |
| anteaya | so smckee is working on a study tracking git habits | 22:21 |
| *** trown is now known as trown|outttypeww | 22:21 | |
| anteaya | smckee: and we were talking in -dev | 22:21 |
| anteaya | smckee: (s)he is interested in know what other projects use our workflow | 22:22 |
| *** daemontool has quit IRC | 22:22 | |
| *** hdd has quit IRC | 22:22 | |
| anteaya | off the top of my head I had wikimedia and hpe | 22:22 |
| anteaya | anyone have any others? | 22:22 |
| *** Sukhdev has quit IRC | 22:22 | |
| anteaya | in any case I have to go for a walk now | 22:23 |
| anteaya | everyone this is smckee | 22:23 |
| smckee | Thanks again for all your help :) | 22:23 |
| anteaya | smckee: this is everyone | 22:23 |
| *** daemontool has joined #openstack-infra | 22:23 | |
| anteaya | happy to help, good luck with the study | 22:23 |
| *** boris-42 has quit IRC | 22:23 | |
| *** jpr has quit IRC | 22:24 | |
| *** baoli__ has joined #openstack-infra | 22:24 | |
| *** baoli has quit IRC | 22:24 | |
| *** armax has quit IRC | 22:25 | |
| *** mase_x200 has joined #openstack-infra | 22:25 | |
| *** baoli_ has quit IRC | 22:26 | |
| *** isaacb has quit IRC | 22:26 | |
| *** baoli__ has quit IRC | 22:27 | |
| smckee | If anybody is interested in the study that anteaya was talking about, we're aiming to study people who do major Git merges such as (but definitely not limited to) large backports that result in major conflicts. | 22:28 |
| *** baoli has joined #openstack-infra | 22:28 | |
| smckee | Basically, the Git nightmare. I have come across people like this in other projects, but looking at openstack, I am also wondering if certain infra designs might prevent these sorts of issues altogether | 22:28 |
| *** rlandy has joined #openstack-infra | 22:29 | |
| smckee | So if anybody has any info in that general realm, I'd be interested in talking :) | 22:29 |
| *** zz_dimtruck is now known as dimtruck | 22:29 | |
| *** baoli has quit IRC | 22:30 | |
| *** gildub has quit IRC | 22:30 | |
| *** baoli has joined #openstack-infra | 22:30 | |
| *** dkranz has quit IRC | 22:32 | |
| *** regXboi has quit IRC | 22:33 | |
| *** sridhar_ram has quit IRC | 22:33 | |
| *** Sukhdev has joined #openstack-infra | 22:33 | |
| *** sridhar_ram has joined #openstack-infra | 22:33 | |
| *** rockyg has quit IRC | 22:34 | |
| *** Sukhdev has quit IRC | 22:34 | |
| *** otter768 has joined #openstack-infra | 22:35 | |
| *** piet has quit IRC | 22:35 | |
| *** tiswanso has quit IRC | 22:37 | |
| *** xyang1 has quit IRC | 22:39 | |
| *** otter768 has quit IRC | 22:40 | |
| sputnik13 | we've had the following patches stuck and not merging for a few days now https://review.openstack.org/#/c/264603/ https://review.openstack.org/#/c/265536/ https://review.openstack.org/#/c/266049/ https://review.openstack.org/#/c/264632/ https://review.openstack.org/#/c/264594/ | 22:40 |
| sputnik13 | have no idea why they're not merging | 22:41 |
| sputnik13 | can anyone help? | 22:41 |
| *** baoli has quit IRC | 22:42 | |
| *** hdd has joined #openstack-infra | 22:43 | |
| *** baoli has joined #openstack-infra | 22:43 | |
| *** thorst has quit IRC | 22:43 | |
| *** thorst has joined #openstack-infra | 22:44 | |
| *** mase_x200 has quit IRC | 22:45 | |
| *** vivekd has quit IRC | 22:46 | |
| tonyb | jeblair: If you have a few minutes I'd appreciate a review on: https://review.openstack.org/#/c/245525/ | 22:47 |
| *** thorst has quit IRC | 22:48 | |
| tonyb | sputnik13: I looked at the first one, and it needs a rebase. | 22:49 |
| tonyb | sputnik13: look for the little orange circle next to the parent SHA1 | 22:49 |
| clarkb | greghaynes: you think we can attemp image builders late afternoon? | 22:51 |
| tonyb | sputnik13: the others need more careful checking to work out why | 22:52 |
| clarkb | I think crazyness is dying down on my end | 22:52 |
| sputnik13 | tonyb thanks, it looks like 3 of the patches have that, https://review.openstack.org/#/c/265536/ this one for example does not | 22:52 |
| clarkb | smckee: so one thing that happens due to our use of gerrit is that changes tend to be small and logically contained | 22:53 |
| *** rajinir has joined #openstack-infra | 22:53 | |
| *** tzn has joined #openstack-infra | 22:53 | |
| clarkb | smckee: this means that unlike github pull requests that can have massive numbers of commits that get merged into master we have many small changes that individually merge into master | 22:53 |
| tonyb | sputnik13: Yeah I'm not sure about that one (and the others in a similar position) | 22:53 |
| clarkb | smckee: it sort of distributes the problem of dealing with rebases to the authors most familiar and able to deal with rebases in any particular section of the code | 22:53 |
| *** armax has joined #openstack-infra | 22:54 | |
| sputnik13 | tonyb that said, I don't think they're stuck because of the need for rebase, usually merge will take care of things for you unless there's a merge conflict | 22:54 |
| *** baoli has quit IRC | 22:54 | |
| *** randallburt has joined #openstack-infra | 22:54 | |
| sputnik13 | and I was able to rebase using the rebase button, so it doesn't seem like a merge conflict | 22:55 |
| *** baoli has joined #openstack-infra | 22:55 | |
| clarkb | smckee: for backports our stable branch policy is pretty restrictive and doesn't allow features, just high priority bugs and similar which also avoids needing to deal with massive merge conflicts | 22:55 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:55 | |
| randallburt | hi there, anyone from infra have time for a quick question? | 22:56 |
| smckee | OK, that makes sense. Is that policy published anywhere, by any chance? | 22:56 |
| craige | o/ | 22:57 |
| *** tzn has quit IRC | 22:57 | |
| randallburt | So I submitted a fix for one of our Heat template tests https://review.openstack.org/#/c/266519/ and it merged. The blocked job then passed the gate once that fix merged. However, once my original patch was approved, the same error shows up again: https://review.openstack.org/#/c/261114/ | 22:58 |
| clarkb | smckee: yes let me find a link | 22:58 |
| tonyb | sputnik13: no, they're not eligable to move forward without a rebase. | 22:58 |
| openstackgerrit | Merged openstack-dev/pbr: Add libjpeg and liberasurecode for tests https://review.openstack.org/267121 | 22:59 |
| clarkb | smckee: https://wiki.openstack.org/wiki/StableBranch#Stable_branch_policy | 22:59 |
| tonyb | sputnik13: the mergeconflict is in the parent patch not the review that wont merge | 22:59 |
| *** mriedem is now known as mriedem_away | 23:00 | |
| smckee | Awesome, thanks clarkb. I have to meet a student to help them for my class, but I really appreciate your help :) | 23:00 |
| *** smckee has quit IRC | 23:01 | |
| *** armax has quit IRC | 23:01 | |
| clarkb | oh hey it ssomeone at oregon state | 23:01 |
| clarkb | we could totally talk over beers | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add ansible as a dependency https://review.openstack.org/266065 | 23:02 |
| pabelanger | jeblair: Woah, nice! | 23:03 |
| jeblair | jhesketh: ^ there's the rough 'hello world' | 23:03 |
| *** thorst has joined #openstack-infra | 23:04 | |
| jhesketh | jeblair: awesome :-) | 23:04 |
| jeblair | jhesketh: my next task is getting a sensible playbook from the config over to the launcher for you to use | 23:05 |
| jeblair | jhesketh: in the interim, things i haven't tackled yet in the launcher include: having ansible route output to a file and performing the rsync of the git directory to the hosts | 23:05 |
| jeblair | jhesketh: and probably lots of other ansibly things | 23:06 |
| *** ccrouch has quit IRC | 23:06 | |
| *** baoli has quit IRC | 23:06 | |
| *** ccrouch1 has joined #openstack-infra | 23:06 | |
| jeblair | jhesketh: so there should be lots of stuff that can happen there that's not blocked on actually having the job playbook to run the job :) | 23:06 |
| *** baoli has joined #openstack-infra | 23:07 | |
| *** thorst has quit IRC | 23:08 | |
| *** amitgandhinz has quit IRC | 23:11 | |
| *** dcentral has joined #openstack-infra | 23:15 | |
| *** dcentral has quit IRC | 23:16 | |
| randallburt | beuler? | 23:17 |
| *** thorst has joined #openstack-infra | 23:17 | |
| rm_work | is the gate broken ATM? | 23:18 |
| rm_work | seeing this on every one of the lbaas gate tests: | 23:18 |
| rm_work | http://logs.openstack.org/31/267031/1/check/gate-neutron-lbaasv2-dsvm-scenario/d130003/logs/devstacklog.txt.gz#_2016-01-13_22_33_47_971 | 23:18 |
| *** baoli_ has joined #openstack-infra | 23:18 | |
| *** annegentle has quit IRC | 23:18 | |
| rm_work | or is that our issue | 23:18 |
| rm_work | possibly due to the missing package name thing above it? | 23:19 |
| clarkb | randallburt: you will need the fix to merge before it shows up in check tests | 23:19 |
| clarkb | randallburt: check only checks against what is merged, the gate does speculative trees though so you can line them up there if things are working otherwise | 23:19 |
| *** baoli__ has joined #openstack-infra | 23:20 | |
| randallburt | clarkb: but it did merge | 23:20 |
| clarkb | randallburt: but did the fialing job run after it merged? | 23:20 |
| clarkb | rm_work: http://logs.openstack.org/31/267031/1/check/gate-neutron-lbaasv2-dsvm-scenario/d130003/logs/devstacklog.txt.gz#_2016-01-13_22_33_46_827 | 23:20 |
| randallburt | clarkb: yep | 23:20 |
| clarkb | rm_work: thats the actual issue | 23:20 |
| clarkb | rm_work: or right below it | 23:21 |
| randallburt | clarkb: the patch that exposed the bug ran fine yesterday. it was approved today and the bug is back | 23:21 |
| rm_work | clarkb: so is that our issue? | 23:21 |
| rm_work | clarkb: or a bigger devstack/gate issue | 23:21 |
| randallburt | clarkb: checked master and nothing stomped on my fix so I am confused. | 23:21 |
| clarkb | rm_work: I think it is your issue | 23:21 |
| *** baoli has quit IRC | 23:21 | |
| rm_work | hmm | 23:21 |
| clarkb | devstack doesn't run dib by default | 23:21 |
| rm_work | hmmm | 23:22 |
| clarkb | randallburt: I am almost positive it is going to be a timing issue but I hvae to unravel it to be sure | 23:22 |
| randallburt | clarkb: k. would just running recheck till it sorts itself help or hurt? | 23:22 |
| rm_work | clarkb: passed about 3 hours ago, failing now, nothing has merged inbetween I don't think :/ | 23:23 |
| clarkb | well if the fix isn't the fix it will just waste test nodes | 23:23 |
| rm_work | but I will look into it | 23:23 |
| *** ociuhandu has joined #openstack-infra | 23:23 | |
| *** baoli_ has quit IRC | 23:23 | |
| randallburt | clarkb: the fix should be the fix since the test for the original patch ran fine yesterday | 23:23 |
| *** peristeri has quit IRC | 23:23 | |
| clarkb | oh ok the fix is to project-config | 23:23 |
| randallburt | clarkb: yep | 23:23 |
| clarkb | so ya its going to be a timing issue I bet | 23:23 |
| clarkb | because puppet has to run then jjb hsa to run and we hva eto do it across 8 jenkins masters | 23:24 |
| clarkb | let me check the configs on the jenkins masters now | 23:24 |
| randallburt | clarkb: ah, so just lucked out on the node selection yesterday? | 23:24 |
| randallburt | clarkb: cool, thanks! | 23:24 |
| clarkb | well its eventually consistent | 23:24 |
| clarkb | so you raced it | 23:24 |
| clarkb | I am going to confirm that it was just slow and not broken though | 23:25 |
| randallburt | clarkb: k | 23:25 |
| clarkb | hrm jenkins01 still has the old config | 23:25 |
| *** amotoki has joined #openstack-infra | 23:28 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Temporarily make lbaasv2-minimal job non-voting, due to breakage https://review.openstack.org/267223 | 23:28 |
| *** baoli__ has quit IRC | 23:29 | |
| dougwig | clarkb: yes, that lbaas breakage looks like a gate hook break. see ^^ to get neutron sorted while they debug it. | 23:29 |
| *** gordc has quit IRC | 23:29 | |
| jhesketh | jeblair: so without having looked at your new changes very closely yet, it looks like you've done parts of the 'create a zuul launcher' task which I thought I was working on? | 23:30 |
| jhesketh | (not that it matters because I hadn't started it yet) | 23:30 |
| *** ZZelle_ has quit IRC | 23:30 | |
| *** Qiming has joined #openstack-infra | 23:31 | |
| clarkb | I am doing a manual job update on 01. 04 will need it too and 06 is dead in the water | 23:32 |
| clarkb | I am going to guess puppet timed out on the exec | 23:33 |
| clarkb | oh no there is an error | 23:33 |
| clarkb | the cache is corrupt | 23:34 |
| clarkb | I am going t oremove the cache | 23:34 |
| *** topol has quit IRC | 23:34 | |
| *** gildub has joined #openstack-infra | 23:34 | |
| *** armax has joined #openstack-infra | 23:37 | |
| *** topol_ has joined #openstack-infra | 23:37 | |
| *** topol_ is now known as Guest57135 | 23:37 | |
| clarkb | it is doing stuff now, the broken cache is in the .cache dir under a new name | 23:38 |
| clarkb | if we want to figure out how it broke | 23:38 |
| jeblair | jhesketh: very little -- just an outline; i'm turning it over to you now :) | 23:38 |
| randallburt | clarkb: cool, thanks for sorting it. I'll try a recheck | 23:38 |
| clarkb | zaro: zxiiro what do you think about maybe having jjb rebuild the cache on its own if it is broken? log an error/warning then move on | 23:38 |
| clarkb | randallburt: well not yet | 23:39 |
| clarkb | randallburt: it hasn't applied the update yet | 23:39 |
| randallburt | clarkb: oh, whoops. | 23:39 |
| clarkb | it is just in progress | 23:39 |
| jeblair | jhesketh: but i had some ideas about how it should vaguely work that i thought getting into code would be easiest | 23:39 |
| jhesketh | jeblair: fair enough | 23:39 |
| jhesketh | I had some different ideas :p | 23:39 |
| *** chlong-afk has quit IRC | 23:40 | |
| clarkb | 8118 jobs generated | 23:40 |
| jhesketh | jeblair: I was thinking nothing would change with the launcher.. but rather we'd just have a 'zuul-worker' or similar binary that would listen to gearman and do the work from there | 23:40 |
| jhesketh | jeblair: that way the zuul-workers can scale similar to the zuul-mergers | 23:40 |
| zxiiro | clarkb: I think that'd be useful. Currently we have a separate weekly job that clears the cache as we've noticed on on rare occasions the cache needs to be cleared to update jobs. | 23:41 |
| jeblair | jhesketh: i think you'll find that's more or less what my change does | 23:41 |
| jeblair | jhesketh: except that the zuul-ansible-launcher contains a merger | 23:41 |
| zaro | clarkb: how would you tell that it's broken? | 23:42 |
| jeblair | jhesketh: mergers must be integral to the launcher now so that they can push git repo state to their workers | 23:42 |
| clarkb | zaro: if loading it throws an exception | 23:42 |
| clarkb | zaro: it is invalid yaml so it explodes | 23:42 |
| jhesketh | jeblair: ah cool, I hadn't read it properly yet, but it looked like it was replacing the gearman launcher | 23:42 |
| jeblair | jhesketh: ah, no, i just renamed the gearman launcher because the filenames were getting out of hand | 23:42 |
| gtmanfred | Shrews: is it possible to rerun jenkins? cause the failures were on code that didn't change between patchset 4 and 5, with 5 only being the addition of the unittest | 23:42 |
| gtmanfred | https://review.openstack.org/#/c/266532/ | 23:42 |
| jhesketh | jeblair: okay and there's just no entry point yet to run the workers? | 23:43 |
| *** Qiming has quit IRC | 23:43 | |
| *** Daisy has joined #openstack-infra | 23:43 | |
| jhesketh | (except in the tests) | 23:43 |
| zaro | clarkb: there could be other reasons for it being broken though. so in that use case it might make sense. | 23:43 |
| jeblair | jhesketh: correct, it only works in unit tests because i explicitly instantiated it. but there should be a real entry point script. the framework is just like a merge worker | 23:43 |
| clarkb | zaro: I mean, its a cache, if it breaks we can safely remove it | 23:43 |
| jhesketh | jeblair: okay cool, looks like we're on the same page :-) | 23:44 |
| jeblair | w00t | 23:44 |
| clarkb | it makes the update run longer but thats probably better than trying to repair it by hand | 23:44 |
| jeblair | jhesketh: it's all yours now, i'll stay out of there and work on the job config stuff for a while | 23:44 |
| jhesketh | jeblair: so I'll review the outstanding v3 patches today.. Did you want me to just go ahead and merge these new ones similar to how we discussed previously and iterate by committing on top? | 23:44 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: 10 day count is too high https://review.openstack.org/267230 | 23:44 |
| jeblair | jhesketh: yeah, i think so | 23:45 |
| jhesketh | jeblair: thanks, I have some catching up to do with the remaining reviews (the last lot took some time) | 23:45 |
| *** dimtruck is now known as zz_dimtruck | 23:46 | |
| jeblair | jhesketh: i really wanted to get an end-to-end demonstration of most of the big features as a skeleton -- and that's there now. | 23:46 |
| *** Swami has quit IRC | 23:46 | |
| zaro | clarkb: true but might be real annoying if it's to aggressive trying to repair. | 23:47 |
| *** akshai has quit IRC | 23:47 | |
| clarkb | same issue on 04, so rerunning there with cache moved aside | 23:47 |
| *** weshay_xchat has quit IRC | 23:48 | |
| jhesketh | jeblair: yeah, it's cool stuff :-) | 23:49 |
| clarkb | zaro: well you don't agressively repair you just remove it and start over | 23:49 |
| anteaya | dougwig: reviewed | 23:50 |
| anteaya | dougwig: also why does neutron have so few jobs in the gate pipeline compared to the check pipeline | 23:50 |
| anteaya | dougwig: our policy is that a voting job is in both check and gate | 23:50 |
| anteaya | not just check | 23:50 |
| pabelanger | fungi: thanks for the invite to the summit! | 23:51 |
| *** armax has quit IRC | 23:51 | |
| jeblair | fungi: fwiw, i think i registered -- i was bounced to https://www.openstack.org/summit/austin-2016/?order=complete after doing so, but didn't really get any reassuring feedback (other than the url). | 23:51 |
| *** amotoki has quit IRC | 23:52 | |
| jeblair | i did get a reassuring email, so that's good. | 23:52 |
| *** amotoki has joined #openstack-infra | 23:52 | |
| * anteaya is glad jeblair is reassured | 23:52 | |
| anteaya | and registered | 23:52 |
| jeblair | what about openstack proposal bot? i'm looking for its invite but i don't see it. | 23:52 |
| fungi | jeblair: reassuring email is reassuring | 23:53 |
| anteaya | ha ha ha | 23:53 |
| anteaya | I think script 2.0 removes the bots | 23:53 |
| clarkb | ok and to fix jenkins06 I am going to stop it, remove the slaves fro mthe config.xml and start it | 23:53 |
| anteaya | clarkb: ack | 23:53 |
| jeblair | i will have to see if i can sneak a bot registration past script 2.0 | 23:53 |
| *** pradk has quit IRC | 23:53 | |
| *** amotoki has quit IRC | 23:53 | |
| fungi | jeblair: the new hotness in 267161 supports an --ignore option (and takes multiples of them) or an ignore list in a local config file | 23:53 |
| anteaya | clarkb: thanks for fixing the jenkinsi | 23:53 |
| fungi | jeblair: so proposal bot should not receive an invite this time | 23:54 |
| dougwig | anteaya: we did that because our failure rate is about 12% per job, and we wanted neutron devs to feel that pain, but not the integrated gate. | 23:54 |
| anteaya | dougwig: well that is not in keeping with our policy | 23:54 |
| *** Daisy has quit IRC | 23:54 | |
| *** akshai has joined #openstack-infra | 23:54 | |
| anteaya | I'm all in favour of thumbscrewing neutron devs if neutron drivers think that will yield the best results | 23:55 |
| jeblair | anteaya: we did make some exceptions with neutron as a remedial step | 23:55 |
| anteaya | however I am concerned with the cargo cult behaviour that zuul/layout.yaml does see | 23:55 |
| *** Daisy has joined #openstack-infra | 23:55 | |
| jeblair | it is worth discussing whether that's still necessary | 23:55 |
| anteaya | jeblair: ah I must have missed that decision | 23:55 |
| anteaya | dougwig: do you feel it is still necessary? | 23:55 |
| jeblair | anteaya: it was a long time ago, it may predate project-config even :) | 23:55 |
| dougwig | anteaya: granted, and they were discussed as exceptions, not the rule. | 23:55 |
| anteaya | I belive that | 23:55 |
| randallburt | clarkb: job is running on 03 and is "over the hump", thanks much. | 23:55 |
| jeblair | anteaya: so anyway, carry on -- just wanted to provide history context | 23:56 |
| anteaya | jeblair: I appreciate context, thank you | 23:56 |
| *** rlandy has quit IRC | 23:56 | |
| * anteaya carries on | 23:56 | |
| anteaya | dougwig: okay so an exception has an end date or end criteria, yeah? | 23:56 |
| *** rbrndt__ has quit IRC | 23:57 | |
| fungi | SamYaple: are you around? did you get a summit code from me? the script decided to die on a connection timeout and yours was the last entry in my log so it may have been trying to send that one when it happened (needs to be more robust... guess i know which script i'm revamping next) | 23:57 |
| anteaya | dougwig: so if we are going with the argument of exception, can we discuss some end criteria? | 23:57 |
| mordred | How many previous OpenStack Summits have you attended? | 23:58 |
| mordred | 12 | 23:58 |
| mordred | ugh | 23:58 |
| mordred | I may have been doing this for a while | 23:58 |
| anteaya | ha ha ha | 23:58 |
| *** armax has joined #openstack-infra | 23:58 | |
| anteaya | since the dawn of time the way I understand it | 23:58 |
| fungi | mordred: they didn't have an option for "all of them"? | 23:58 |
| dougwig | anteaya: that's fair. I'd like to tie it to job failure rates. I don't think end criteria have been discussed yet. | 23:58 |
| jeblair | fungi: no, one has to count the alphabet each time | 23:58 |
| clarkb | and 06 is starting up again | 23:58 |
| dougwig | On my phone, so typing is hard. :) | 23:58 |
| anteaya | dougwig: oh okay yes, let's do the philosphical conversation when you have a full keyboard | 23:59 |
| mordred | fungi: I have missed one | 23:59 |
| mordred | fungi: I was not at the Bexar summit | 23:59 |
| fungi | mordred: nowai! | 23:59 |
| anteaya | dougwig: but note I see only two non-template jobs in neutron gate and about a dozen in neutron check and that feels odd to me | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!