| lifeless | wenlock: no | 00:00 |
|---|---|---|
| lifeless | wenlock: what are you trying to figure out :) | 00:00 |
| clarkb | wenlock: if ubuntu counts as debian | 00:00 |
| clarkb | we do have centos slaves too | 00:00 |
| wenlock | lifeless, trying to figure out how to properly start zuul... and deal with it when the server is rebooted | 00:01 |
| fungi | wenlock: what does that have to do with jenkins slave platforms> | 00:01 |
| fungi | ? | 00:01 |
| fungi | (we don't stop and start zuul on slaves) | 00:01 |
| fungi | (we don't even install zuul on slaves) | 00:02 |
| wenlock | fungi: nothing... sorry... was thinking about a different problem | 00:02 |
| wenlock | fungi, correct | 00:02 |
| wenlock | fungi, we combined our slave and master | 00:02 |
| clarkb | wenlock: in jeblair's opinion the proper way is by hand | 00:02 |
| wenlock | fungi, got confused | 00:02 |
| lifeless | wenlock: ok, so what are you trying to figure out by asking about debian slaves? | 00:02 |
| *** virmitio has left #openstack-infra | 00:02 | |
| wenlock | fungi, whole start stop mechanisum on if a server like the master is rebooted | 00:03 |
| fungi | clarkb: nodepool deletes seem to be happening fine still. only 8 nodes in total in that state now | 00:03 |
| clarkb | fungi: great | 00:03 |
| wenlock | fungi, which is correct, not a slave problem | 00:03 |
| *** boris-42_ has quit IRC | 00:03 | |
| fungi | wenlock: sudo update-rc.d zuul defaults | 00:04 |
| *** ekarlso- has joined #openstack-infra | 00:04 | |
| wenlock | fungi, perfect | 00:04 |
| fungi | should make it start at priority 20 on runlevels 2,3,4,5 and stop on 0,1,6 | 00:04 |
| wenlock | fungi, im going to give that a shot | 00:05 |
| clarkb | fungi: wenlock: or just add a service def for it that makes the service enabled but not running | 00:05 |
| clarkb | and add a toggle for that in puppet so that jeblair can have it manual and others can have it automagical | 00:05 |
| fungi | yeah, if you want puppet to configure it, then what clarkb said | 00:06 |
| *** markwash has joined #openstack-infra | 00:06 | |
| wenlock | clarkb:.... trying to understand... why would you try not starting automatically... ? | 00:06 |
| wenlock | fungi.... oooh | 00:06 |
| fungi | wenlock: if you stopped it on purpose | 00:06 |
| wenlock | fungi, yep, another good reason | 00:06 |
| fungi | having it suddenly come back on because puppet||reboot could make badness | 00:06 |
| *** boris-4__ has quit IRC | 00:07 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8 https://review.openstack.org/70865 | 00:08 |
| zaro | clarkb: ^ | 00:09 |
| *** gokrokve has joined #openstack-infra | 00:09 | |
| *** yamahata has joined #openstack-infra | 00:10 | |
| wenlock | carlkb, lifeless, fungi: thanks for clearing that up for me :D | 00:10 |
| *** marun has joined #openstack-infra | 00:10 | |
| wenlock | clarkb: changing the subject... how is it going with reviews for today, i'd like another on https://review.openstack.org/#/c/51425/ | 00:10 |
| wenlock | i know it has evil ruby in it :< | 00:11 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only https://review.openstack.org/70223 | 00:11 |
| jog0 | clarkb: what did I miss this time ^ | 00:11 |
| clarkb | zaro: approved | 00:11 |
| *** matsuhashi has joined #openstack-infra | 00:12 | |
| zaro | clarkb: thanks! | 00:13 |
| jog0 | http://status.openstack.org/zuul/ looks strangly quit -- a lot of queued jobs | 00:14 |
| *** marun has quit IRC | 00:15 | |
| jog0 | very few running | 00:15 |
| clarkb | jog0: you should probably offset the two crons | 00:15 |
| *** alexpilotti has quit IRC | 00:15 | |
| clarkb | jog0: either with like a random dely or with a different cron spec | 00:15 |
| jog0 | how do I do that? | 00:15 |
| clarkb | jog0: if you grep for RANDOM in the infra/config repo you will see other crons that do that | 00:16 |
| clarkb | jog0: yes that does look dodd | 00:16 |
| *** david-lyle has quit IRC | 00:16 | |
| *** pcrews has quit IRC | 00:16 | |
| clarkb | fungi: ^ is nodepool still happy? | 00:16 |
| zaro | clarkb: puppet agent is running on zuul-dev right? | 00:17 |
| clarkb | zaro: uh maybe, I will double check | 00:17 |
| clarkb | zaro: nope, starting it now | 00:18 |
| jog0 | clarkb: hah sleep command => $((RANDOM\%600)) && ...' | 00:18 |
| *** thuc has quit IRC | 00:18 | |
| fungi | jog0: queuing tripleo jobs, or other jobs? | 00:18 |
| clarkb | fungi: other jobs | 00:18 |
| *** thuc has joined #openstack-infra | 00:19 | |
| clarkb | reconfigure managers is failing due to tripleo | 00:19 |
| *** david-lyle has joined #openstack-infra | 00:20 | |
| *** eharney has quit IRC | 00:20 | |
| *** mriedem has joined #openstack-infra | 00:20 | |
| fungi | another place where the code assumes providers will always be online? | 00:20 |
| fungi | i can delete the tripleo cloud back out of the config and restart nodepool again | 00:21 |
| clarkb | fungi: possibly, I think we should try that | 00:21 |
| clarkb | fungi: I don't see the typical node allocation debug messages | 00:21 |
| fungi | i'll stop puppet this time | 00:21 |
| fungi | so it won't be undone | 00:21 |
| jog0 | clarkb: how about doing minute => '*/16', | 00:21 |
| jog0 | so one job runs every 15 min, the othe revery 16 | 00:22 |
| clarkb | jog0: that would work as well but maybe it less clear when the job will run | 00:22 |
| jog0 | clarkb: or i can do | 00:23 |
| jog0 | command => 'sleep $((RANDOM\%60+90)) && | 00:23 |
| jog0 | but don't know what that means | 00:23 |
| *** thuc has quit IRC | 00:23 | |
| fungi | nodepool.yaml edited to remove tripleo provider/images/nodes and nodepoold restarting now | 00:23 |
| clarkb | that means sleep for at least 90 seconds with an additional random 0-59 second sleep | 00:23 |
| jog0 | oh perfect | 00:24 |
| *** dcramer_ has joined #openstack-infra | 00:25 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only https://review.openstack.org/70223 | 00:25 |
| jog0 | clarkb: hopefully I did it right this time | 00:26 |
| *** jaypipes has quit IRC | 00:27 | |
| fungi | there seems to be an inverse correlation between the times when i need nodepoold's graceful restart to return quickly, and the times when it actually does | 00:29 |
| clarkb | hahaha | 00:29 |
| clarkb | it knows | 00:29 |
| *** yamahata has quit IRC | 00:29 | |
| *** mikal_ is now known as mikal | 00:29 | |
| fungi | if this takes much longer, i'll just kill it and clean up any mess it leaves behind | 00:32 |
| *** jaypipes has joined #openstack-infra | 00:32 | |
| *** nosnos has joined #openstack-infra | 00:33 | |
| *** yamahata has joined #openstack-infra | 00:33 | |
| fungi | i think it's waiting for that image build to complete | 00:34 |
| lifeless | fungi: yeah, it doesn't know how to suspend-restore that | 00:35 |
| lifeless | we should fix that | 00:35 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1270212 https://review.openstack.org/70386 | 00:38 |
| *** masayukig has joined #openstack-infra | 00:39 | |
| *** thuc has joined #openstack-infra | 00:39 | |
| *** yamahata has quit IRC | 00:43 | |
| *** kraman has joined #openstack-infra | 00:43 | |
| *** nosnos_ has joined #openstack-infra | 00:44 | |
| *** masayukig has quit IRC | 00:44 | |
| clarkb | fungi: how are we doing? | 00:46 |
| lifeless | anteaya: devananda: markmcclain: https://review.openstack.org/#/c/70549/ has landed; neutron in theory is no longer wedged. | 00:47 |
| *** nosnos has quit IRC | 00:47 | |
| markmcclain | lifeless: thanks | 00:48 |
| devananda | lifeless: ack. i have several rechecks going in ironic to confirm if we're unwedged | 00:48 |
| fungi | clarkb: i marked all the building images and nodes for deletion and then killed and restarted nodepoold | 00:48 |
| *** yamahata has joined #openstack-infra | 00:48 | |
| markmcclain | same here I've issued a recheck on a critical review | 00:48 |
| clarkb | fungi: I see allocation requests now in debug log which should mean we will get nodes | 00:48 |
| *** oubiwann_ has joined #openstack-infra | 00:48 | |
| fungi | i'll keep an eye out for unusually old ready/used/delete nodes here in a little while and delete them if they stick around | 00:49 |
| *** ekarlso has quit IRC | 00:50 | |
| lifeless | markmcclain: OOI which review? | 00:51 |
| *** dcramer_ has quit IRC | 00:51 | |
| markmcclain | https://review.openstack.org/67537 | 00:51 |
| *** weshay has joined #openstack-infra | 00:52 | |
| clarkb | fungi: 273 nodes building | 00:52 |
| clarkb | fungi: did we lose that image build in the restart? | 00:52 |
| fungi | clarkb: yeah, its started again now | 00:52 |
| lifeless | markmcclain: fingers crossed | 00:53 |
| *** nosnos_ has quit IRC | 00:54 | |
| *** nosnos has joined #openstack-infra | 00:54 | |
| *** gokrokve has quit IRC | 00:55 | |
| *** david_lyle_ has joined #openstack-infra | 00:55 | |
| *** CaptTofu has joined #openstack-infra | 00:56 | |
| *** gyee has quit IRC | 00:59 | |
| *** david-lyle has quit IRC | 00:59 | |
| *** mrodden has quit IRC | 01:00 | |
| clarkb | fungi: it looks like zuul and nodepool are almost completely caught up | 01:03 |
| *** dcramer_ has joined #openstack-infra | 01:03 | |
| *** atiwari has quit IRC | 01:03 | |
| *** thuc has quit IRC | 01:10 | |
| *** thuc has joined #openstack-infra | 01:10 | |
| fungi | excellent | 01:10 |
| clarkb | tripleo being the only the only oustanding nodepool/zuul trouble | 01:11 |
| *** wenlock has quit IRC | 01:13 | |
| *** thuc has quit IRC | 01:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make requirement update proposals more robust. https://review.openstack.org/69969 | 01:15 |
| *** mrodden has joined #openstack-infra | 01:15 | |
| markmcclain | lifeless: no luck… https://jenkins01.openstack.org/job/check-tempest-dsvm-neutron-pg/910/console | 01:20 |
| lifeless | markmcclain: thats what the others were doing? | 01:22 |
| lifeless | markmcclain: or is this a different pathology? | 01:22 |
| clarkb | http://logs.openstack.org/37/67537/3/check/check-tempest-dsvm-neutron-pg/da51a31/logs/screen-n-cpu.txt.gz?level=ERROR | 01:22 |
| markmcclain | guestfs failures | 01:22 |
| clarkb | looks the same | 01:22 |
| lifeless | so thats bong | 01:22 |
| clarkb | lifeless: perhaps the libvirt driver is buggy? | 01:23 |
| * clarkb looks at it again | 01:23 | |
| lifeless | 2014-02-04 00:57:07.087 | 2014-02-04 00:57:07 + iniset /etc/nova/nova.conf libvirt inject_partition -2 | 01:23 |
| clarkb | ya I think the value was set properly | 01:23 |
| *** amcrn has quit IRC | 01:24 | |
| lifeless | I can't see any override of it either | 01:25 |
| *** mestery has joined #openstack-infra | 01:25 | |
| clarkb | but according to the logs it somehow fell into the block after elif inject_files and CONF.libvirt.inject_partition != -2: | 01:25 |
| *** mestery has quit IRC | 01:26 | |
| *** sarob has quit IRC | 01:26 | |
| *** mestery has joined #openstack-infra | 01:26 | |
| lifeless | if ( if any((key, net, metadata, admin_pass, files)): | 01:27 |
| lifeless | but thats guarded bu | 01:27 |
| lifeless | *by( | 01:27 |
| lifeless | bah | 01:27 |
| lifeless | *by* | 01:27 |
| lifeless | elif inject_files and CONF.libvirt.inject_partition != -2: | 01:27 |
| lifeless | git grep inject_partition in devstack only has one hit, the setting line | 01:28 |
| clarkb | lifeless: is it possible that the file is overridden after the ini set? | 01:28 |
| *** marun has joined #openstack-infra | 01:28 | |
| clarkb | lifeless: yup that is the problem :/ | 01:29 |
| clarkb | see the location of create_nova_conf | 01:29 |
| lifeless | yes | 01:29 |
| lifeless | just spotted that | 01:29 |
| lifeless | this is a broader bug | 01:31 |
| lifeless | that whole function is toast | 01:31 |
| clarkb | ya | 01:32 |
| clarkb | the cgruop stuff is probably ok | 01:32 |
| clarkb | now I feel bad saying that hte devstack change would override the nova changes | 01:33 |
| clarkb | I was very wrong about that :0 | 01:33 |
| *** marun has quit IRC | 01:33 | |
| lifeless | well | 01:34 |
| lifeless | we thought it would :P | 01:34 |
| lifeless | chmouel: dtroyer: sdague: https://review.openstack.org/70885 | 01:34 |
| lifeless | chmouel: dtroyer sdague: russell's change was broken, my change was broken. This fixes the breakage so it should actually take effect. | 01:35 |
| lifeless | Of course, that may cause it to break in other ways :/ | 01:35 |
| *** masayukig has joined #openstack-infra | 01:40 | |
| *** david-lyle has joined #openstack-infra | 01:40 | |
| *** david_lyle_ has quit IRC | 01:44 | |
| *** masayukig has quit IRC | 01:45 | |
| *** ryanpetrello has quit IRC | 01:46 | |
| *** mestery has quit IRC | 01:47 | |
| *** gokrokve has joined #openstack-infra | 01:47 | |
| *** pcrews has joined #openstack-infra | 01:49 | |
| *** sandywalsh_ has quit IRC | 01:50 | |
| *** ken1ohmichi has joined #openstack-infra | 01:50 | |
| *** jecarey has joined #openstack-infra | 01:50 | |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/requirements: HTTPretty: bump to 0.8.0 https://review.openstack.org/70886 | 01:53 |
| *** markmcclain1 has joined #openstack-infra | 01:54 | |
| *** markmcclain has quit IRC | 01:54 | |
| *** markmcclain1 has quit IRC | 01:56 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 01:59 |
| *** nati_uen_ has joined #openstack-infra | 01:59 | |
| *** nati_uen_ has quit IRC | 01:59 | |
| *** gokrokve has quit IRC | 02:00 | |
| *** nati_uen_ has joined #openstack-infra | 02:00 | |
| *** markmcclain has joined #openstack-infra | 02:01 | |
| *** CaptTofu has quit IRC | 02:02 | |
| *** nati_ueno has quit IRC | 02:02 | |
| *** russellb has joined #openstack-infra | 02:02 | |
| *** ryanpetrello has joined #openstack-infra | 02:02 | |
| *** CaptTofu has joined #openstack-infra | 02:03 | |
| *** gokrokve has joined #openstack-infra | 02:03 | |
| *** shivh has left #openstack-infra | 02:03 | |
| *** sarob has joined #openstack-infra | 02:04 | |
| *** sandywalsh_ has joined #openstack-infra | 02:06 | |
| *** masayukig has joined #openstack-infra | 02:09 | |
| portante | jog0: you around? | 02:11 |
| portante | I'd like to chat about that path I should take for those two e-r checks I have in review | 02:11 |
| jog0 | portante: yup | 02:12 |
| jog0 | portante: pulling 'em up now | 02:12 |
| jog0 | so https://review.openstack.org/#/c/69702/2/queries/1270608.yaml | 02:13 |
| portante | so for the IPv6 one, should I just fix up the duplicate, and then open another bug for the wider Network is unreachable match? | 02:13 |
| jog0 | I want to see an example | 02:13 |
| portante | yes, I'll post the two logstash queries for that | 02:13 |
| jog0 | portante: yeah, although I *thought* I opened a bug for the unable to install pip part | 02:13 |
| jog0 | but perhaps I imagined it | 02:13 |
| portante | i ran them, then yanked down the export with build_uuid only and uniq'd them to show that the new is a complete superset of the old | 02:14 |
| *** crank has quit IRC | 02:14 | |
| jog0 | so we should still do the rename for https://review.openstack.org/#/c/69670/ | 02:14 |
| jog0 | so if you want to propose a patch to just do that I will +A right now | 02:14 |
| portante | jog0: can you find that bug for me? | 02:14 |
| portante | okay, I'll do that now, sec | 02:14 |
| *** crank has joined #openstack-infra | 02:14 | |
| jog0 | portante: sure let me look | 02:14 |
| *** oubiwan__ has joined #openstack-infra | 02:15 | |
| portante | I'll propose a new patch for the straight rename, sec | 02:16 |
| *** oubiwann has quit IRC | 02:16 | |
| *** oubiwann_ has quit IRC | 02:16 | |
| *** oubiwann has joined #openstack-infra | 02:18 | |
| *** melwitt has quit IRC | 02:18 | |
| jog0 | portante: https://bugs.launchpad.net/openstack-ci/+bug/1270710 | 02:18 |
| portante | gotit, thanks, jog0 | 02:18 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Update fingerprint to match opened bug 1266616 https://review.openstack.org/70887 | 02:21 |
| portante | that is the straight rename | 02:21 |
| portante | So bug 1270710 does not talk about the Network is unreachable | 02:23 |
| portante | do you want to keep that out of it and just open another bug to cover the more generic NiU? | 02:23 |
| portante | jog0 | 02:23 |
| * portante workings on the log stash queries for the other one out for review | 02:24 | |
| jog0 | portante: hmm yeah that was a timeout | 02:24 |
| lifeless | clarkb: so 70885 passed everything (except bash8 which i"m fixing) | 02:24 |
| lifeless | clarkb: but I think all the relevant people are asleep | 02:24 |
| jog0 | but my hunch is its the same underlying issue (but I could be very wrong) | 02:24 |
| *** sarob has quit IRC | 02:25 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Update fingerprint to match opened bug 1266616 https://review.openstack.org/70887 | 02:25 |
| *** sarob has joined #openstack-infra | 02:25 | |
| clarkb | lifeless: :( ya sdague is in korea and dtroyer probably hit his EOD and jeblair is flying back to the US (I think) | 02:25 |
| clarkb | fungi: you still about? | 02:25 |
| clarkb | chmouel: vishy: either of you guys able to review and possibly approve a devstack change? | 02:26 |
| fungi | clarkb: yep. mass deleting the cruft nodes from before the restart now (any in a used state for more than 3 hours) | 02:27 |
| *** oubiwan__ has quit IRC | 02:27 | |
| portante | jog0: do you want to go with your hunch and update the bug to talk about the Network is unreachable? | 02:27 |
| portante | those are two different patterns, so it might be worth while keeping them separate until we can confirm the actually share a root cause | 02:27 |
| *** oubiwann_ has joined #openstack-infra | 02:28 | |
| clarkb | fungi: 70885 is something we should probably try to get in, but there is a lack of devstack cores around. I have a feeling we can get away with the current broken state until tomorrow morning though | 02:28 |
| clarkb | considering it has been broken for a little bit now | 02:28 |
| *** david-lyle has quit IRC | 02:29 | |
| *** sarob has quit IRC | 02:30 | |
| jog0 | portante: keep 'em different for now | 02:30 |
| lifeless | clarkb: what I'd really like is to see the neutron/ironic jobs that fail tested against that change | 02:30 |
| lifeless | clarkb: could we perhaps add them as non-voting-on-devstack but run-on-devstack ? | 02:31 |
| lifeless | clarkb: then we don't need to wait for it to land to know... | 02:31 |
| fungi | check experimental--i think they're there already | 02:31 |
| * lifeless waits for browser to come back from sync() | 02:32 | |
| lifeless | this may take a while | 02:32 |
| lifeless | fungi: I don't think so | 02:33 |
| lifeless | fungi: only one jjob | 02:33 |
| lifeless | fungi: neutron-heat-slow | 02:33 |
| lifeless | which doesn't say isolated... | 02:33 |
| lifeless | and the definition doesn't have DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION=1 set | 02:34 |
| lifeless | so it won't tell us | 02:34 |
| fungi | oh, i must have been thinking about tempest | 02:34 |
| clarkb | I thought ironic already does non voting against devstack? | 02:34 |
| clarkb | devananda: ^ did we not add it there? | 02:34 |
| clarkb | I am completely on board with adding those jobs to devstack check | 02:35 |
| lifeless | clarkb: ironic is there | 02:35 |
| portante | jog0: regarding the iSCSI one, I have just posted the two log stash queries as inline comments so you can compare them yourself to see what I am talking about | 02:35 |
| lifeless | clarkb: and a devstack-dsvm-neutron | 02:35 |
| portante | I could be off on this | 02:35 |
| lifeless | I don't know what the config of that is yet | 02:35 |
| * lifeless pokes | 02:35 | |
| clarkb | lifeless: what do we need to do then? | 02:35 |
| *** ryanpetrello has quit IRC | 02:35 | |
| portante | jog0: I'll file a separate bug for the generic "Network is unreachable" under openstack-ci and add an e-r query for it | 02:36 |
| fungi | clarkb: still need to test the neutron-isolated job against it | 02:36 |
| lifeless | clarkb: pipeline}-devstack-dsvm-neutron{branch-designator doesn't have tenant isolation on either | 02:37 |
| lifeless | at least, in my copy of /config | 02:38 |
| clarkb | it doesn't the isolated jobs are different | 02:39 |
| lifeless | how do you make something vote in one place but not another ? | 02:39 |
| clarkb | but do we need the extra jobs to see if this fixes the problem? | 02:39 |
| lifeless | well there are two cases we want to fix | 02:39 |
| lifeless | ironic and neutron | 02:39 |
| lifeless | it's possible that this will fix one and not the other | 02:39 |
| lifeless | only one of the cases are represented in jobs on devstack today. | 02:40 |
| clarkb | lifeless: you can't directly. Instead we overload branch-designator to create a second job with an identical config and make that job nonvoting. the ironic -nv jobs do this | 02:40 |
| clarkb | lifeless: oh are we using config drive on the normal neutron tests? | 02:40 |
| *** rakhmerov has quit IRC | 02:40 | |
| clarkb | I thought the neutron postgres test was using file injection and failing too | 02:40 |
| lifeless | clarkb: I don't know the details, but the normal ones pass | 02:40 |
| *** rakhmerov has joined #openstack-infra | 02:40 | |
| *** rakhmerov has quit IRC | 02:40 | |
| lifeless | the specific interaction cared about only shows up in -isolated- jobs | 02:40 |
| *** masayuki_ has joined #openstack-infra | 02:40 | |
| lifeless | AIUI | 02:41 |
| lifeless | the devstack-dsvm job did fail, looking for a cause | 02:41 |
| lifeless | failed on floating_ips | 02:41 |
| lifeless | which is the thing I was asking anteaya about the other day | 02:41 |
| lifeless | (I still haven't filed a bug for that, but I believe its disabled in neutron gate tests as known broke) | 02:42 |
| *** oubiwann_ has quit IRC | 02:43 | |
| *** oubiwann_ has joined #openstack-infra | 02:44 | |
| *** masayuki_ has quit IRC | 02:45 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add Neutron isolated tenants to ds/experimental https://review.openstack.org/70889 | 02:46 |
| lifeless | clarkb: something like that perhaps | 02:46 |
| lifeless | clarkb: season to taste | 02:46 |
| clarkb | that works | 02:46 |
| lifeless | if we can get that live on zuul a check experimental should tell us what we need | 02:47 |
| lifeless | assuming I understood everything :P | 02:48 |
| *** nati_ueno has joined #openstack-infra | 02:48 | |
| clarkb | yup | 02:48 |
| clarkb | I approved it | 02:48 |
| *** nati_ueno has quit IRC | 02:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Neutron isolated tenants to ds/experimental https://review.openstack.org/70889 | 02:49 |
| *** nati_ueno has joined #openstack-infra | 02:50 | |
| markmcclain | lifeless: you beat me too it, I was about to suggest we add the isolated job | 02:50 |
| * markmcclain is low on battery will check in after landing | 02:50 | |
| *** nati_uen_ has quit IRC | 02:51 | |
| lifeless | clarkb: let me know when to do check experimental | 02:53 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1275994 https://review.openstack.org/70892 | 02:54 |
| clarkb | lifeless: 10 after the hour is safest, there is work to add the last layout reload time to the zuul status page which will make this easier in the future | 02:55 |
| portante | jog0: above is the new e-r tracking the new bug, and I put references between the bugs so we can keep them associated | 02:55 |
| jog0 | portante: thanks | 02:55 |
| jog0 | I'll review in the AM | 02:55 |
| portante | k, 'night | 02:55 |
| *** markmcclain has quit IRC | 02:56 | |
| *** markwash has quit IRC | 02:59 | |
| *** oubiwann_ has quit IRC | 03:00 | |
| *** dkehn_ has joined #openstack-infra | 03:03 | |
| *** nati_ueno has quit IRC | 03:11 | |
| *** russellb has quit IRC | 03:12 | |
| *** boris-42_ has joined #openstack-infra | 03:12 | |
| *** nati_ueno has joined #openstack-infra | 03:12 | |
| *** nati_ueno has quit IRC | 03:13 | |
| *** nati_ueno has joined #openstack-infra | 03:13 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Allow webob 1.3 https://review.openstack.org/70741 | 03:14 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client https://review.openstack.org/70897 | 03:19 |
| lifeless | devananda: your ironic run still failed, but not due to guestfs anymore | 03:19 |
| lifeless | devananda: with 70885 | 03:20 |
| *** krotscheck has quit IRC | 03:21 | |
| jog0 | lifeless: http://status.openstack.org/zuul/ tripleo jobs are backed up | 03:24 |
| lifeless | jog0: I know | 03:25 |
| lifeless | jog0: see waaay back backscroll | 03:25 |
| lifeless | jog0: look for muppet. | 03:25 |
| jog0 | lifeless: tripleo weding nodepool and whatnot | 03:25 |
| *** gokrokve has quit IRC | 03:26 | |
| *** sarob has joined #openstack-infra | 03:26 | |
| * lifeless goes to pick up C | 03:26 | |
| *** gokrokve has joined #openstack-infra | 03:26 | |
| *** gokrokve has quit IRC | 03:30 | |
| *** malini is now known as malini_afk | 03:32 | |
| *** gokrokve has joined #openstack-infra | 03:32 | |
| *** sarob has quit IRC | 03:32 | |
| *** emagana has quit IRC | 03:39 | |
| *** gokrokve has quit IRC | 03:40 | |
| *** jecarey has quit IRC | 03:41 | |
| *** masayuki_ has joined #openstack-infra | 03:41 | |
| *** masayuki_ has quit IRC | 03:46 | |
| *** david-lyle has joined #openstack-infra | 03:57 | |
| *** dcramer_ has quit IRC | 04:01 | |
| *** sandywalsh_ has quit IRC | 04:02 | |
| *** gokrokve has joined #openstack-infra | 04:11 | |
| *** gokrokve_ has joined #openstack-infra | 04:13 | |
| *** harlowja is now known as harlowja_away | 04:13 | |
| *** harlowja_away is now known as harlowja | 04:15 | |
| *** koolhead17 has quit IRC | 04:16 | |
| *** gokrokve has quit IRC | 04:16 | |
| *** sandywalsh_ has joined #openstack-infra | 04:17 | |
| *** gokrokve_ has quit IRC | 04:19 | |
| *** sarob has joined #openstack-infra | 04:21 | |
| *** pcrews has quit IRC | 04:23 | |
| *** jgriffith has quit IRC | 04:24 | |
| *** mriedem has quit IRC | 04:24 | |
| *** AlexF has joined #openstack-infra | 04:26 | |
| *** jgriffith has joined #openstack-infra | 04:29 | |
| *** koolhead17 has joined #openstack-infra | 04:31 | |
| *** CaptTofu has quit IRC | 04:32 | |
| *** gokrokve has joined #openstack-infra | 04:32 | |
| *** AlexF has quit IRC | 04:37 | |
| *** ryanpetrello has joined #openstack-infra | 04:39 | |
| *** ryanpetrello has quit IRC | 04:40 | |
| *** masayuki_ has joined #openstack-infra | 04:42 | |
| *** nati_ueno has quit IRC | 04:44 | |
| StevenK | That's one gate reset too many. Can someone promote 70891,2 to unblock the nova gate? | 04:46 |
| *** masayuki_ has quit IRC | 04:47 | |
| *** emagana has joined #openstack-infra | 04:49 | |
| *** emagana has quit IRC | 04:55 | |
| *** thuc has joined #openstack-infra | 04:58 | |
| *** ryanpetrello has joined #openstack-infra | 05:00 | |
| *** amcrn has joined #openstack-infra | 05:02 | |
| lifeless | clarkb: fungi: ^ | 05:03 |
| *** sarob has quit IRC | 05:04 | |
| *** DinaBelova_ is now known as DinaBelova | 05:04 | |
| fungi | StevenK: lifeless: done | 05:04 |
| *** sarob has joined #openstack-infra | 05:04 | |
| StevenK | fungi: Thanks! | 05:05 |
| *** sarob_ has joined #openstack-infra | 05:07 | |
| *** sarob has quit IRC | 05:09 | |
| *** dims has quit IRC | 05:11 | |
| *** sarob has joined #openstack-infra | 05:13 | |
| *** dims has joined #openstack-infra | 05:13 | |
| *** sarob_ has quit IRC | 05:16 | |
| *** thuc has quit IRC | 05:19 | |
| *** slong has joined #openstack-infra | 05:19 | |
| *** thuc has joined #openstack-infra | 05:19 | |
| *** slong_ has quit IRC | 05:20 | |
| *** coolsvap has joined #openstack-infra | 05:21 | |
| *** dkliban has quit IRC | 05:22 | |
| *** amotoki has joined #openstack-infra | 05:22 | |
| *** thuc has quit IRC | 05:23 | |
| *** gokrokve has quit IRC | 05:24 | |
| *** Ryan_Lane has joined #openstack-infra | 05:24 | |
| *** rakhmerov has joined #openstack-infra | 05:25 | |
| *** rakhmerov has joined #openstack-infra | 05:25 | |
| *** markmcclain has joined #openstack-infra | 05:28 | |
| *** sarob has quit IRC | 05:32 | |
| *** sarob has joined #openstack-infra | 05:32 | |
| *** gokrokve has joined #openstack-infra | 05:33 | |
| zaro | clarkb: so i see you already tried recheck on review-dev. seems like it's not working, do you happen to know whether the project-dev.yaml got updated? | 05:34 |
| clarkb | zaro: I havent tried since you patched the layout file | 05:35 |
| zaro | clarkb: ohh yeah, didn't notice your rechecks were from a few days ago. i just tried but zuul didn't trigger another job run. | 05:36 |
| clarkb | zaro: Oh I bet we need to make the regex multiline | 05:37 |
| clarkb | because of the newlines | 05:37 |
| *** sarob has quit IRC | 05:37 | |
| *** gokrokve has quit IRC | 05:38 | |
| zaro | clarkb: actually it didn't get merged. crap, i made that change dependent on https://review.openstack.org/#/c/60348 | 05:38 |
| *** markwash has joined #openstack-infra | 05:40 | |
| zaro | that was a mistake making it dependent on 60348. what would you suggest i do? | 05:40 |
| clarkb | remove the dependdncy | 05:41 |
| *** rwsu has quit IRC | 05:41 | |
| zaro | you mean push new patch without the dependency? | 05:41 |
| clarkb | ya | 05:41 |
| * zaro tries that | 05:42 | |
| *** masayuki_ has joined #openstack-infra | 05:43 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8 https://review.openstack.org/70865 | 05:47 |
| *** masayuki_ has quit IRC | 05:47 | |
| zaro | clarkb: ^ | 05:48 |
| *** slong has quit IRC | 05:50 | |
| *** AlexF has joined #openstack-infra | 05:54 | |
| *** AlexF has quit IRC | 05:59 | |
| *** AlexF has joined #openstack-infra | 05:59 | |
| *** nosnos_ has joined #openstack-infra | 06:00 | |
| *** ryanpetrello has quit IRC | 06:00 | |
| *** markwash has quit IRC | 06:02 | |
| *** nosnos has quit IRC | 06:03 | |
| *** sarob has joined #openstack-infra | 06:03 | |
| *** jcoufal has joined #openstack-infra | 06:04 | |
| *** AlexF has quit IRC | 06:04 | |
| *** VijayT has quit IRC | 06:07 | |
| openstackgerrit | Bill Maxwell proposed a change to openstack-infra/jenkins-job-builder: added logstash plugin support and fixed test typo https://review.openstack.org/70917 | 06:07 |
| *** AlexF has joined #openstack-infra | 06:08 | |
| *** hdd has quit IRC | 06:10 | |
| *** pcrews has joined #openstack-infra | 06:16 | |
| zaro | clarkb: it was a trivial rebase, no approval this time? | 06:20 |
| *** e0ne has joined #openstack-infra | 06:23 | |
| *** AlexF has quit IRC | 06:24 | |
| clarkb | zaro trivial rebase does not preserve approvals. I can push it through in the morning | 06:24 |
| *** jcooley_ has joined #openstack-infra | 06:26 | |
| *** e0ne has quit IRC | 06:27 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs https://review.openstack.org/70594 | 06:29 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs https://review.openstack.org/70594 | 06:30 |
| *** Ryan_Lane1 has joined #openstack-infra | 06:30 | |
| *** sarob has quit IRC | 06:31 | |
| *** resker has joined #openstack-infra | 06:31 | |
| *** pcrews has quit IRC | 06:32 | |
| *** Shrews_ has joined #openstack-infra | 06:32 | |
| *** coolsvap_away has joined #openstack-infra | 06:32 | |
| *** matsuhashi has quit IRC | 06:33 | |
| *** gokrokve has joined #openstack-infra | 06:33 | |
| *** matsuhashi has joined #openstack-infra | 06:33 | |
| *** branen_ has joined #openstack-infra | 06:33 | |
| *** SpamapS_ has joined #openstack-infra | 06:33 | |
| *** soren has joined #openstack-infra | 06:34 | |
| *** soren has quit IRC | 06:34 | |
| *** soren has joined #openstack-infra | 06:34 | |
| *** masayuki_ has joined #openstack-infra | 06:35 | |
| *** flaper87l has joined #openstack-infra | 06:35 | |
| *** flaper87l is now known as flaper87 | 06:35 | |
| lifeless | fungi: / clarkb: are you going to reenable tripleo cloudbefore you sleep ? | 06:35 |
| *** jd__` has joined #openstack-infra | 06:37 | |
| *** gokrokve has quit IRC | 06:38 | |
| *** Ryan_Lane has quit IRC | 06:38 | |
| *** coolsvap has quit IRC | 06:38 | |
| *** dims has quit IRC | 06:38 | |
| *** oubiwann has quit IRC | 06:38 | |
| *** masayukig has quit IRC | 06:38 | |
| *** mattoliverau has quit IRC | 06:38 | |
| *** jhesketh__ has quit IRC | 06:38 | |
| *** primeministerp has quit IRC | 06:38 | |
| *** SpamapS has quit IRC | 06:38 | |
| *** soren_ has quit IRC | 06:38 | |
| *** w_ has quit IRC | 06:38 | |
| *** Shrews has quit IRC | 06:38 | |
| *** bodepd has quit IRC | 06:38 | |
| *** esker has quit IRC | 06:38 | |
| *** branen has quit IRC | 06:38 | |
| *** zul has quit IRC | 06:38 | |
| *** med_ has quit IRC | 06:38 | |
| *** katyafervent has quit IRC | 06:38 | |
| *** flaper87|afk has quit IRC | 06:38 | |
| *** jd__ has quit IRC | 06:38 | |
| *** krtaylor has quit IRC | 06:38 | |
| *** mordred has quit IRC | 06:38 | |
| *** matrohon has quit IRC | 06:38 | |
| *** jd__` is now known as jd__ | 06:38 | |
| *** primeministerp has joined #openstack-infra | 06:40 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs https://review.openstack.org/70594 | 06:42 |
| *** mrda is now known as mrda_away | 06:43 | |
| *** masayukig has joined #openstack-infra | 06:44 | |
| *** harlowja is now known as harlowja_away | 06:44 | |
| *** katyafervent has joined #openstack-infra | 06:44 | |
| *** denis_makogon_ has joined #openstack-infra | 06:45 | |
| *** mattoliverau has joined #openstack-infra | 06:45 | |
| *** zul has joined #openstack-infra | 06:46 | |
| juice | if anyone has time to review this patch that will help trove gate get up and running, we'd appreciate it https://review.openstack.org/#/c/70905/ | 06:47 |
| *** masayukig has quit IRC | 06:48 | |
| *** skraynev_ is now known as skraynev | 06:50 | |
| *** bhuvan has joined #openstack-infra | 06:51 | |
| *** mordred has joined #openstack-infra | 06:54 | |
| *** oubiwann has joined #openstack-infra | 06:54 | |
| *** jhesketh_ has joined #openstack-infra | 06:55 | |
| *** bodepd has joined #openstack-infra | 06:56 | |
| *** VijayT has joined #openstack-infra | 06:57 | |
| *** krtaylor has joined #openstack-infra | 06:59 | |
| *** olaph has joined #openstack-infra | 07:01 | |
| *** bhuvan has quit IRC | 07:02 | |
| *** coolsvap_away has quit IRC | 07:04 | |
| *** bhuvan has joined #openstack-infra | 07:04 | |
| *** jkt_ is now known as jkt | 07:05 | |
| *** dims has joined #openstack-infra | 07:08 | |
| *** odyssey4me has joined #openstack-infra | 07:08 | |
| *** markmcclain has quit IRC | 07:10 | |
| *** VijayT has quit IRC | 07:12 | |
| *** jcooley_ has quit IRC | 07:12 | |
| *** saju_m has joined #openstack-infra | 07:12 | |
| *** bhuvan has quit IRC | 07:15 | |
| *** lcheng_ has joined #openstack-infra | 07:16 | |
| openstackgerrit | Lukas Bednar proposed a change to openstack-infra/jenkins-job-builder: scms: MultiSCM is generated for scms hidden in macro https://review.openstack.org/69827 | 07:18 |
| *** slong has joined #openstack-infra | 07:19 | |
| *** flaper87 has quit IRC | 07:21 | |
| *** flaper87 has joined #openstack-infra | 07:21 | |
| *** yamahata has quit IRC | 07:23 | |
| *** coolsvap has joined #openstack-infra | 07:26 | |
| *** dstufft is now known as dstufft-gone | 07:26 | |
| *** talluri has joined #openstack-infra | 07:27 | |
| *** yolanda has joined #openstack-infra | 07:28 | |
| *** AlexF has joined #openstack-infra | 07:32 | |
| *** gokrokve has joined #openstack-infra | 07:33 | |
| *** gokrokve has quit IRC | 07:38 | |
| *** AlexF has quit IRC | 07:41 | |
| *** rwsu has joined #openstack-infra | 07:42 | |
| *** sarob has joined #openstack-infra | 07:43 | |
| *** masayukig has joined #openstack-infra | 07:44 | |
| *** lcheng_ has quit IRC | 07:47 | |
| *** amcrn has quit IRC | 07:47 | |
| *** sarob has quit IRC | 07:48 | |
| *** jcooley_ has joined #openstack-infra | 07:48 | |
| *** pblaho has joined #openstack-infra | 07:48 | |
| *** masayukig has quit IRC | 07:48 | |
| *** slong has quit IRC | 07:49 | |
| *** rwsu has quit IRC | 07:51 | |
| *** rwsu has joined #openstack-infra | 07:53 | |
| *** jcooley_ has quit IRC | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:59 | |
| *** afazekas has joined #openstack-infra | 08:01 | |
| *** kruskakli has joined #openstack-infra | 08:04 | |
| *** ken1ohmichi has left #openstack-infra | 08:05 | |
| *** afazekas has quit IRC | 08:07 | |
| *** denis_makogon_ has quit IRC | 08:09 | |
| *** sarob has joined #openstack-infra | 08:13 | |
| *** che-arne has quit IRC | 08:14 | |
| openstackgerrit | Steven Hardy proposed a change to openstack/requirements: Update python-keystoneclient to 0.5.0 https://review.openstack.org/70929 | 08:15 |
| *** sarob has quit IRC | 08:18 | |
| *** afazekas has joined #openstack-infra | 08:23 | |
| *** fbo is now known as fbo_away | 08:32 | |
| *** gokrokve has joined #openstack-infra | 08:33 | |
| *** pblaho has quit IRC | 08:34 | |
| *** pblaho has joined #openstack-infra | 08:35 | |
| *** coolsvap has quit IRC | 08:36 | |
| *** gokrokve has quit IRC | 08:37 | |
| *** ogelbukh1 is now known as ogelbukh | 08:42 | |
| *** yamahata has joined #openstack-infra | 08:43 | |
| *** masayukig has joined #openstack-infra | 08:45 | |
| *** hashar has joined #openstack-infra | 08:46 | |
| *** masayukig has quit IRC | 08:50 | |
| *** ilyashakhat has joined #openstack-infra | 08:55 | |
| *** jpich has joined #openstack-infra | 08:56 | |
| *** slong has joined #openstack-infra | 09:02 | |
| *** jhesketh_ has quit IRC | 09:02 | |
| *** jhesketh has quit IRC | 09:02 | |
| *** bhuvan has joined #openstack-infra | 09:03 | |
| chmouel | clarkb: sure, is it still needed? | 09:04 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add replication key for gerrit from hiera https://review.openstack.org/70933 | 09:05 |
| chmouel | clarkb: ok i have approved 70885 | 09:06 |
| *** markmc has joined #openstack-infra | 09:13 | |
| *** sarob has joined #openstack-infra | 09:13 | |
| *** jhesketh has joined #openstack-infra | 09:14 | |
| *** jhesketh_ has joined #openstack-infra | 09:15 | |
| openstackgerrit | Fei Long Wang proposed a change to openstack/requirements: Depending on python-swiftclient>=1.6 https://review.openstack.org/70935 | 09:16 |
| *** sarob has quit IRC | 09:18 | |
| *** masayuki_ has quit IRC | 09:20 | |
| *** fbo_away is now known as fbo | 09:22 | |
| *** coolsvap has joined #openstack-infra | 09:23 | |
| *** johnthetubaguy has joined #openstack-infra | 09:24 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to test_remote https://review.openstack.org/70936 | 09:27 |
| *** jooools has joined #openstack-infra | 09:28 | |
| *** johnthetubaguy has quit IRC | 09:31 | |
| *** johnthetubaguy has joined #openstack-infra | 09:32 | |
| *** gokrokve has joined #openstack-infra | 09:33 | |
| *** Ryan_Lane1 has quit IRC | 09:36 | |
| *** gokrokve has quit IRC | 09:38 | |
| *** e0ne has quit IRC | 09:46 | |
| *** masayukig has joined #openstack-infra | 09:46 | |
| *** e0ne has joined #openstack-infra | 09:46 | |
| *** katyafervent is now known as katyafervent_awa | 09:49 | |
| *** reed has joined #openstack-infra | 09:50 | |
| *** e0ne_ has joined #openstack-infra | 09:50 | |
| *** e0ne has quit IRC | 09:50 | |
| *** masayukig has quit IRC | 09:51 | |
| *** matrohon has joined #openstack-infra | 09:51 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot. https://review.openstack.org/70672 | 09:52 |
| *** BobBallAway is now known as BobBall | 09:52 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot. https://review.openstack.org/70672 | 09:59 |
| *** e0ne_ has quit IRC | 10:02 | |
| *** e0ne has joined #openstack-infra | 10:02 | |
| *** e0ne has quit IRC | 10:06 | |
| *** e0ne has joined #openstack-infra | 10:07 | |
| *** VijayT has joined #openstack-infra | 10:09 | |
| *** e0ne has quit IRC | 10:10 | |
| *** e0ne has joined #openstack-infra | 10:10 | |
| *** sarob has joined #openstack-infra | 10:13 | |
| *** VijayT has quit IRC | 10:14 | |
| *** jcooley_ has joined #openstack-infra | 10:17 | |
| *** sarob has quit IRC | 10:18 | |
| *** Adri2000_ is now known as Adri2000 | 10:21 | |
| *** lcestari has joined #openstack-infra | 10:22 | |
| *** metabro has quit IRC | 10:22 | |
| *** jcooley_ has quit IRC | 10:23 | |
| *** coolsvap has quit IRC | 10:30 | |
| *** gokrokve has joined #openstack-infra | 10:33 | |
| openstackgerrit | Alexandre Levine proposed a change to openstack-infra/config: Removing python26 jobs for gce-api project https://review.openstack.org/70944 | 10:35 |
| mordred | just so that everyone knows ... | 10:35 |
| mordred | I'd really like it if I could configure my phone to turn on usb tethering every time it's plugged in somewhere | 10:36 |
| *** gokrokve has quit IRC | 10:37 | |
| bogdando | fungi, hi. sorry for delayed reply. >>> <fungi> bogdando: if you have a particular change/bug combination which you believe meets those criteria, i can take a look and try to determine what happened | 10:43 |
| *** boris-42_ has quit IRC | 10:43 | |
| bogdando | fungi, please take a look onto https://bugs.launchpad.net/fuel/+bug/1271176 or later bugs from Fuel project | 10:43 |
| bogdando | fungi, I've had to change status manually after status tracking were lost | 10:44 |
| *** slong has quit IRC | 10:44 | |
| *** dizquierdo has joined #openstack-infra | 10:46 | |
| *** masayukig has joined #openstack-infra | 10:46 | |
| *** cody-somerville has joined #openstack-infra | 10:46 | |
| *** e0ne has quit IRC | 10:50 | |
| *** e0ne has joined #openstack-infra | 10:50 | |
| *** masayukig has quit IRC | 10:51 | |
| *** che-arne has joined #openstack-infra | 10:52 | |
| *** hashar has quit IRC | 10:58 | |
| *** russellb has joined #openstack-infra | 11:00 | |
| *** DinaBelova is now known as DinaBelova_ | 11:01 | |
| *** CaptTofu has joined #openstack-infra | 11:02 | |
| *** DinaBelova_ is now known as DinaBelova | 11:02 | |
| *** CaptTofu has quit IRC | 11:07 | |
| *** DinaBelova is now known as DinaBelova_ | 11:10 | |
| *** e0ne has quit IRC | 11:10 | |
| *** jcooley_ has joined #openstack-infra | 11:13 | |
| *** sarob has joined #openstack-infra | 11:13 | |
| *** andre__ has quit IRC | 11:14 | |
| *** sarob_ has joined #openstack-infra | 11:16 | |
| *** sarob has quit IRC | 11:18 | |
| *** jcooley_ has quit IRC | 11:19 | |
| *** sarob_ has quit IRC | 11:20 | |
| *** jcoufal has quit IRC | 11:26 | |
| *** andre__ has joined #openstack-infra | 11:29 | |
| *** david-lyle has quit IRC | 11:29 | |
| *** nosnos_ has quit IRC | 11:30 | |
| *** CaptTofu has joined #openstack-infra | 11:32 | |
| *** gokrokve has joined #openstack-infra | 11:33 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: pbr: update to 0.6 https://review.openstack.org/70955 | 11:35 |
| *** gokrokve has quit IRC | 11:38 | |
| *** amotoki has quit IRC | 11:38 | |
| *** amotoki has joined #openstack-infra | 11:38 | |
| *** masayukig has joined #openstack-infra | 11:47 | |
| *** rfolco has joined #openstack-infra | 11:48 | |
| *** masayukig has quit IRC | 11:52 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:58 | |
| *** miqui has quit IRC | 12:04 | |
| *** yamahata has quit IRC | 12:04 | |
| *** malini_afk is now known as malini | 12:06 | |
| *** yamahata has joined #openstack-infra | 12:09 | |
| *** sarob has joined #openstack-infra | 12:13 | |
| *** sandywalsh_ has quit IRC | 12:15 | |
| *** esker has joined #openstack-infra | 12:17 | |
| *** e0ne has joined #openstack-infra | 12:17 | |
| *** sarob has quit IRC | 12:18 | |
| *** resker has quit IRC | 12:20 | |
| *** yassine has joined #openstack-infra | 12:20 | |
| *** exz has joined #openstack-infra | 12:22 | |
| *** CaptTofu has quit IRC | 12:22 | |
| *** e0ne_ has joined #openstack-infra | 12:24 | |
| *** e0ne has quit IRC | 12:24 | |
| *** bhuvan has quit IRC | 12:28 | |
| *** sandywalsh_ has joined #openstack-infra | 12:28 | |
| *** e0ne has joined #openstack-infra | 12:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Allow webob 1.3 https://review.openstack.org/70741 | 12:29 |
| *** e0ne__ has joined #openstack-infra | 12:30 | |
| *** e0ne___ has joined #openstack-infra | 12:31 | |
| *** e0ne__ has quit IRC | 12:31 | |
| *** boris-42_ has joined #openstack-infra | 12:32 | |
| *** e0ne_ has quit IRC | 12:32 | |
| *** e0ne___ has quit IRC | 12:32 | |
| *** masayukig has joined #openstack-infra | 12:32 | |
| *** e0ne_ has joined #openstack-infra | 12:32 | |
| *** gokrokve has joined #openstack-infra | 12:33 | |
| *** vogxn has joined #openstack-infra | 12:34 | |
| *** e0ne has quit IRC | 12:34 | |
| *** matsuhashi has quit IRC | 12:36 | |
| *** masayukig has quit IRC | 12:36 | |
| *** matsuhashi has joined #openstack-infra | 12:36 | |
| *** yamahata has quit IRC | 12:37 | |
| *** gokrokve has quit IRC | 12:37 | |
| *** DinaBelova_ is now known as DinaBelova | 12:38 | |
| *** matsuhashi has quit IRC | 12:41 | |
| *** bhuvan has joined #openstack-infra | 12:46 | |
| *** smarcet has joined #openstack-infra | 12:46 | |
| *** hashar has joined #openstack-infra | 12:49 | |
| *** yamahata has joined #openstack-infra | 12:51 | |
| *** DinaBelova is now known as DinaBelova_ | 12:52 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add xtrigger. https://review.openstack.org/70971 | 12:53 |
| *** DinaBelova_ is now known as DinaBelova | 12:55 | |
| *** DinaBelova is now known as DinaBelova_ | 12:55 | |
| *** weshay has quit IRC | 12:56 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: Implements: blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts https://review.openstack.org/69636 | 12:56 |
| *** markmcclain has joined #openstack-infra | 12:59 | |
| *** DinaBelova_ is now known as DinaBelova | 13:01 | |
| *** yamahata has quit IRC | 13:03 | |
| *** e0ne has joined #openstack-infra | 13:05 | |
| *** russellb_ has joined #openstack-infra | 13:06 | |
| *** russellb_ has quit IRC | 13:07 | |
| *** yamahata has joined #openstack-infra | 13:07 | |
| *** jhesketh_ has quit IRC | 13:07 | |
| *** jhesketh has quit IRC | 13:08 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: Implements: blueprint openid-oauth2-infra-implementation-documentation https://review.openstack.org/69620 | 13:08 |
| *** AlexF has joined #openstack-infra | 13:08 | |
| *** e0ne_ has quit IRC | 13:08 | |
| *** DinaBelova is now known as DinaBelova_ | 13:10 | |
| *** e0ne_ has joined #openstack-infra | 13:13 | |
| *** sarob has joined #openstack-infra | 13:13 | |
| *** bhuvan has quit IRC | 13:14 | |
| *** e0ne has quit IRC | 13:16 | |
| *** sarob has quit IRC | 13:17 | |
| *** alexpilotti has joined #openstack-infra | 13:18 | |
| *** bhuvan has joined #openstack-infra | 13:18 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Merger: log non Git exceptions on merge https://review.openstack.org/70977 | 13:20 |
| *** jcoufal has joined #openstack-infra | 13:21 | |
| *** CaptTofu has joined #openstack-infra | 13:22 | |
| *** yamahata has quit IRC | 13:26 | |
| *** rainya has joined #openstack-infra | 13:27 | |
| *** annegentle has joined #openstack-infra | 13:27 | |
| *** pvo has joined #openstack-infra | 13:27 | |
| *** jbryce has joined #openstack-infra | 13:27 | |
| *** lillie has joined #openstack-infra | 13:27 | |
| *** smarcet has quit IRC | 13:28 | |
| *** lillie is now known as Guest46806 | 13:28 | |
| *** talluri has quit IRC | 13:29 | |
| *** talluri has joined #openstack-infra | 13:30 | |
| *** weshay has joined #openstack-infra | 13:32 | |
| *** yamahata has joined #openstack-infra | 13:32 | |
| *** masayukig has joined #openstack-infra | 13:33 | |
| *** gokrokve has joined #openstack-infra | 13:33 | |
| *** talluri has quit IRC | 13:34 | |
| *** miqui has joined #openstack-infra | 13:36 | |
| *** masayukig has quit IRC | 13:37 | |
| *** gokrokve has quit IRC | 13:38 | |
| *** saju_m has quit IRC | 13:41 | |
| *** smarcet has joined #openstack-infra | 13:42 | |
| *** ekarlso- has quit IRC | 13:45 | |
| *** ekarlso has joined #openstack-infra | 13:45 | |
| *** yamahata has quit IRC | 13:46 | |
| *** ryanpetrello has joined #openstack-infra | 13:46 | |
| chmouel | mordred: should be possible with tasker ? (if a plugin detect what usb host it connected to or via gps location) | 13:46 |
| *** yamahata has joined #openstack-infra | 13:50 | |
| *** leifmadsen has quit IRC | 13:50 | |
| *** leifmadsen has joined #openstack-infra | 13:50 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/requirements: Add a new dependency: trollius https://review.openstack.org/70983 | 13:53 |
| *** yamahata has quit IRC | 13:55 | |
| *** prad_ has joined #openstack-infra | 13:55 | |
| *** rossella_s has joined #openstack-infra | 13:56 | |
| *** thomasem has joined #openstack-infra | 13:56 | |
| *** yamahata has joined #openstack-infra | 13:59 | |
| *** dims has quit IRC | 14:03 | |
| *** dkranz has quit IRC | 14:05 | |
| *** dprince has joined #openstack-infra | 14:05 | |
| *** beagles is now known as beagles_brb | 14:06 | |
| *** Shrews_ is now known as Shrews | 14:06 | |
| *** mattoliverau has quit IRC | 14:09 | |
| *** hashar has quit IRC | 14:10 | |
| *** hashar has joined #openstack-infra | 14:10 | |
| *** sarob has joined #openstack-infra | 14:13 | |
| *** malini is now known as malini_afk | 14:13 | |
| *** mriedem has joined #openstack-infra | 14:14 | |
| *** markmcclain has quit IRC | 14:15 | |
| *** vogxn has quit IRC | 14:15 | |
| *** vogxn has joined #openstack-infra | 14:16 | |
| *** sarob has quit IRC | 14:18 | |
| *** yamahata has quit IRC | 14:18 | |
| *** yamahata has joined #openstack-infra | 14:19 | |
| fungi | lifeless: is the tripleo cloud back online? if so i'll start puppet back up on the nodepool server | 14:19 |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Adding query for bug 1275500 https://review.openstack.org/70756 | 14:20 |
| *** hashar_ has joined #openstack-infra | 14:20 | |
| *** thuc has joined #openstack-infra | 14:20 | |
| *** hashar has quit IRC | 14:21 | |
| *** pblaho has quit IRC | 14:22 | |
| *** hashar_ is now known as hashar | 14:23 | |
| *** changbl has quit IRC | 14:23 | |
| *** prad_ has quit IRC | 14:24 | |
| *** andreaf has joined #openstack-infra | 14:29 | |
| *** vogxn has quit IRC | 14:29 | |
| *** oubiwann_ has joined #openstack-infra | 14:32 | |
| fungi | bogdando: looks like we have a bug with the update script, probably resulting from recent changes to the library we use to interpret our projects lists | 14:32 |
| fungi | mordred: http://paste.openstack.org/show/62443 (any idea what we missed?) | 14:33 |
| fungi | i'll go ahead and open a bug | 14:33 |
| *** gokrokve has joined #openstack-infra | 14:33 | |
| *** kraman has quit IRC | 14:33 | |
| *** masayukig has joined #openstack-infra | 14:33 | |
| *** amotoki has quit IRC | 14:35 | |
| fungi | filed bug 1276180 for this | 14:36 |
| *** masayukig has quit IRC | 14:38 | |
| *** gokrokve has quit IRC | 14:38 | |
| *** thuc has quit IRC | 14:39 | |
| *** thuc has joined #openstack-infra | 14:40 | |
| *** medberry has joined #openstack-infra | 14:40 | |
| *** medberry has quit IRC | 14:40 | |
| *** medberry has joined #openstack-infra | 14:40 | |
| *** coolsvap has joined #openstack-infra | 14:42 | |
| *** thuc has quit IRC | 14:44 | |
| *** wchrisj_ has quit IRC | 14:48 | |
| *** malini_afk is now known as malini | 14:48 | |
| *** jcooley_ has joined #openstack-infra | 14:51 | |
| *** jecarey has joined #openstack-infra | 14:53 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Toplevel collection typecheck. https://review.openstack.org/70997 | 14:54 |
| *** DinaBelova_ is now known as DinaBelova | 14:54 | |
| *** boris-42_ has quit IRC | 14:54 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add xtrigger. https://review.openstack.org/70971 | 14:54 |
| *** jcooley_ has quit IRC | 14:57 | |
| bogdando | fungi, thank you for the fast update | 14:57 |
| fungi | bogdando: now let's see whether i fix it before someone else beats me to it ;) | 14:58 |
| *** mfer has joined #openstack-infra | 15:00 | |
| *** dkliban has joined #openstack-infra | 15:02 | |
| *** jasondotstar has joined #openstack-infra | 15:04 | |
| *** matrohon has quit IRC | 15:05 | |
| *** dcramer_ has joined #openstack-infra | 15:08 | |
| jeblair | fungi: good morning! | 15:10 |
| fungi | it's a jeblair! [and the crowd roars] | 15:11 |
| *** krotscheck has joined #openstack-infra | 15:12 | |
| jeblair | fungi: let me know if there's anything urgent that needs my attention | 15:12 |
| mordred | chmouel: thanks - I will investigate taht | 15:13 |
| chmouel | mordred: i think lifehacker had a long series about that app and how to automate everything on an android phone with it | 15:13 |
| *** sarob has joined #openstack-infra | 15:13 | |
| mordred | jeblair: you did not contract my illness did you? | 15:14 |
| mordred | jeblair: gothicmindfood and krotscheck each got a little I think | 15:14 |
| mordred | chmouel: I think I remember a friend of mine trying to tell me it was awesome at some point | 15:14 |
| krotscheck | .... where "a little" means a fever of 102 for going on 2 days. | 15:15 |
| fungi | jeblair: nothing super urgent. we have puppet disabled on nodepool again because another extended tripleo provider outage exposed additional bugs we haven't sorted out yet (in short, restarting nodepool with one configured provider completely offline was causing reconfigure managers to fail. i'm waiting for confirmation that the provider is operational again before firing puppet back up | 15:15 |
| jeblair | mordred, krotscheck: oh no! i'm sorry to hear that! | 15:15 |
| krotscheck | jeblair, mordred: Did y'all get a chance to lok at the UI mocks for the MVP yet? | 15:16 |
| mordred | krotscheck: was just about to - and yes, it sounds like you got what I got. | 15:16 |
| *** dkranz has joined #openstack-infra | 15:16 | |
| mordred | krotscheck: it is not pleasant is it? | 15:16 |
| jeblair | krotscheck: not yet; i'm just rejoining the land of the connected | 15:17 |
| krotscheck | Here's the link: http://docs-draft.openstack.org/97/70897/1/check/gate-storyboard-webclient-js-unittests/32bcc1d/dist/ | 15:17 |
| krotscheck | jcoufal ^ | 15:17 |
| *** AlexF has quit IRC | 15:17 | |
| *** sarob has quit IRC | 15:18 | |
| krotscheck | mordred: It's REALLY not pleasant. The fever seems to be dropping though. | 15:18 |
| jeblair | fungi: ack thanks. yeah, it's probably time for nodepool to grow up a bit. | 15:18 |
| mordred | krotscheck: I'm assuming you also have the chest, nose and aching? | 15:18 |
| *** hashar has quit IRC | 15:19 | |
| krotscheck | mordred: No, mine's all sinus, all the time. | 15:19 |
| jeblair | mordred: i believe you have described "the flu". | 15:19 |
| *** hashar has joined #openstack-infra | 15:19 | |
| jcoufal | krotscheck: nice job, I'll have a look on that (maybe not today but tomorrow morning) | 15:19 |
| jcoufal | krotscheck: take a rest... | 15:19 |
| mordred | jeblair: well, less GI than I'd normally expect with fly | 15:20 |
| mordred | flu | 15:20 |
| jeblair | krotscheck: sinus+fever is usually suspected to be bacterial so you can probably get antibiotics if you see a doc | 15:21 |
| *** jnoller has joined #openstack-infra | 15:22 | |
| krotscheck | jeblair: I think my snot's the wrong color for that.... | 15:22 |
| * krotscheck continues down the TMI train | 15:22 | |
| *** dizquierdo_afk is now known as dizquierdo | 15:22 | |
| krotscheck | jcoufal: When you do take a look, check the mobile sizing as well. | 15:23 |
| dansmith | I have a weird PEP8 failure complaining that I didn't update the nova.conf.sample, but I didn't touch anything that would warrant it: | 15:23 |
| dansmith | https://review.openstack.org/#/c/70215 | 15:23 |
| *** eharney has joined #openstack-infra | 15:23 | |
| fungi | dansmith: last time this cropped up, it was from an oslo sync which merged ahead of the failing changes | 15:24 |
| jcoufal | krotscheck: absolutely | 15:24 |
| dansmith | fungi: how could that happen? | 15:24 |
| jeblair | krotscheck: color isn't a great indicator; usually anything non-clear is indicative. (/me just had viral(probably) sinusitis and so is rather familiar with this from the negative side) | 15:25 |
| krotscheck | So it's YOUR fault! | 15:25 |
| fungi | dansmith: i'm not quite sure what methods the api sample comparison routine checks, so i don't really know... that was just the explanation i saw in the post-mortem | 15:26 |
| fungi | dansmith: have you tried regenerating the samples to see what the diff ends up being? | 15:26 |
| jeblair | krotscheck: yes, my viral sinusitis from 2 weeks ago caused your bacterial sinusitis. :) i'm just a walking bio-lab. | 15:26 |
| dansmith | fungi: okay, this is unrelated to api samples, this is the nova.conf sample | 15:26 |
| dansmith | fungi: and I don't see how something could have merged that introduced a change and didn't update the sample, as this should have caught them | 15:26 |
| markmc | dansmith, I'd say that's coming from keystone | 15:27 |
| jeblair | krotscheck: blame mordred's flu too | 15:27 |
| fungi | ahh... could changes to oslo.config affect that? | 15:27 |
| dhellmann | dansmith: see https://bugs.launchpad.net/nova/+bug/1268614 | 15:27 |
| jeblair | i'm going to let the people talking about work talk about work now. :) | 15:27 |
| dansmith | dhellmann: ah, thanks | 15:27 |
| markmc | right, when we extract options from a library | 15:27 |
| dhellmann | dansmith: that example was a change in keystoneclient | 15:27 |
| markmc | and the library adds new options | 15:27 |
| markmc | nova is now taking options from both keystoneclient and oslo.messaging | 15:28 |
| markmc | adding options to either will break nova | 15:28 |
| dansmith | yep, I got it now | 15:28 |
| dhellmann | and that list is going to grow as we graduate more libraries | 15:28 |
| dhellmann | so we need to rethink this particular test | 15:28 |
| dansmith | see, this is why I didn't even check the rechecks list.. in my early-morning pre-coffee stuper, I thought "for sure I have a unique bug here!" | 15:28 |
| mordred | we MAY want to use zuuls ability to do coordinated gating | 15:29 |
| markmc | check should probably check that this particular commit changes the sample file | 15:29 |
| markmc | as opposed to checking whether the file is out of date | 15:29 |
| fungi | maybe the sample config generation wants to become a patch proposal job? | 15:29 |
| *** gokrokve has joined #openstack-infra | 15:29 | |
| mordred | so that for specific things where there is potential for multi-project gating, we do them ina combined context | 15:29 |
| dhellmann | fungi: that was proposed, as was building the conf file as part of packaging | 15:29 |
| dhellmann | it feels weird to be checking in a file we generate | 15:30 |
| *** bhuvan has quit IRC | 15:30 | |
| markmc | mordred, so, if you add a config option to oslo.messaging you need to simultaneously submit patches to all projects which use it? | 15:30 |
| *** gokrokve_ has joined #openstack-infra | 15:30 | |
| markmc | dhellmann, yeah, it does | 15:30 |
| mordred | markmc: ah - no, nevermind | 15:30 |
| mordred | markmc: I shall now shut up | 15:30 |
| markmc | mordred, heh, np :) | 15:30 |
| dhellmann | how is this file used? is it just for docs, or do real people use it as the basis for deployments? | 15:31 |
| *** bhuvan has joined #openstack-infra | 15:31 | |
| markmc | dhellmann, for RDO, AFAIK we make it the default config file installed | 15:32 |
| markmc | dhellmann, distro defaults (like "use qpid", or "keep state in /var/lib") goes in /usr/share/nova/nova-dist.conf | 15:32 |
| dhellmann | I don't know enough about distro packaging -- could we just provide a script to generate the file and not ship the file itself? | 15:33 |
| dhellmann | i.e., "install all of these packages and run foo.sh" | 15:33 |
| *** pcrews has joined #openstack-infra | 15:33 | |
| *** gokrokve has quit IRC | 15:33 | |
| markmc | yeah, we can do that | 15:33 |
| *** hashar has quit IRC | 15:33 | |
| markmc | so long as we keep a lid on the dependencies required to generate the sample file | 15:34 |
| *** david-lyle has joined #openstack-infra | 15:34 | |
| *** kraman has joined #openstack-infra | 15:34 | |
| markmc | e.g. allowing the config file to be generated without redis installed | 15:34 |
| dhellmann | ok, there's a mailing list thread on this ("pep8 gating fails due to tools/config/check_uptodate.sh") so maybe we should discuss it there | 15:34 |
| markmc | yeah | 15:34 |
| dhellmann | yeah, that's already supposed to be a requirement for plugins | 15:34 |
| markmc | yep | 15:34 |
| *** masayukig has joined #openstack-infra | 15:34 | |
| dhellmann | and we can move the generate script into oslo.config so apps only need to provide a wrapper to pass all the right args | 15:34 |
| *** jgrimm_ has joined #openstack-infra | 15:35 | |
| *** wenlock has joined #openstack-infra | 15:36 | |
| markmc | dhellmann, | 15:38 |
| markmc | http://lists.openstack.org/pipermail/openstack-dev/2014-February/026253.html | 15:38 |
| openstackgerrit | Mauro Rodrigues proposed a change to openstack-infra/devstack-gate: Remove exercises tests from grenade runs https://review.openstack.org/71010 | 15:39 |
| *** masayukig has quit IRC | 15:39 | |
| dhellmann | markmc: ack | 15:40 |
| *** hashar has joined #openstack-infra | 15:45 | |
| *** jcooley_ has joined #openstack-infra | 15:46 | |
| *** krotscheck has quit IRC | 15:50 | |
| *** krotscheck has joined #openstack-infra | 15:50 | |
| *** DinaBelova is now known as DinaBelova_ | 15:51 | |
| *** beagles_brb is now known as beagles | 15:51 | |
| *** jcooley_ has quit IRC | 15:51 | |
| *** amotoki has joined #openstack-infra | 15:53 | |
| *** DinaBelova_ is now known as DinaBelova | 15:53 | |
| *** oubiwann_ has quit IRC | 15:54 | |
| openstackgerrit | A change was merged to openstack/requirements: HTTPretty: bump to 0.8.0 https://review.openstack.org/70886 | 15:55 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot. https://review.openstack.org/70672 | 15:57 |
| *** CaptTofu has quit IRC | 15:57 | |
| *** oubiwann_ has joined #openstack-infra | 15:58 | |
| *** atiwari has joined #openstack-infra | 16:01 | |
| *** CaptTofu has joined #openstack-infra | 16:02 | |
| *** mrodden has quit IRC | 16:03 | |
| fungi | jeblair: speaking of nodepool fun, that ^ patch is a workaround to an interesting paramiko side-effect | 16:03 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 16:05 |
| *** krotscheck is now known as krotscheck_sick | 16:07 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot. https://review.openstack.org/70672 | 16:08 |
| *** bhuvan has quit IRC | 16:09 | |
| *** jcooley_ has joined #openstack-infra | 16:09 | |
| *** bhuvan has joined #openstack-infra | 16:10 | |
| devananda | g'morning, folks | 16:12 |
| openstackgerrit | Matt Farina proposed a change to openstack-infra/gitdm: Adding Matt Farina. https://review.openstack.org/70422 | 16:13 |
| *** emagana has joined #openstack-infra | 16:13 | |
| fungi | devananda: i don't suppose you happen to have any status information for the tripleo-ci cloud lifeless was rebuilding yesterday? | 16:13 |
| devananda | fungi: i do not. however, ironic's gate is still broken | 16:13 |
| devananda | fungi: i heard that folks landed some sort of fix overnight. it seems to have merely changed the way our gate is broken | 16:14 |
| fungi | devananda: oh, did the devstack patch which corrected the previous patch make it in? last i saw they were trying to find devstack cores still awake to approve it | 16:14 |
| devananda | fungi: i believe so | 16:15 |
| malini | fungi: hello again!! | 16:15 |
| devananda | i haven't had any coffee yet, and without its super powers, i can not even find where the failure in https://review.openstack.org/#/c/70869/ is | 16:15 |
| fungi | devananda: yeah, https://review.openstack.org/70885 is the one i was thinking of, and it's merged as of 12:30 utc | 16:16 |
| fungi | malini: howdy | 16:16 |
| *** mrodden has joined #openstack-infra | 16:16 | |
| malini | fungi: good :) I have another patch set ready for you https://review.openstack.org/#/c/65140/5/ | 16:16 |
| malini | Can you chk when you get a chace? | 16:16 |
| *** bhuvan has quit IRC | 16:17 | |
| fungi | malini: i think you still had a problem with the rebase on that change | 16:20 |
| fungi | see yje review comment i just added | 16:20 |
| fungi | s/yje/the/ | 16:20 |
| *** prad_ has joined #openstack-infra | 16:20 | |
| fungi | devananda: i see the non-voting and experimental checks which ran for 70885 didn't fix the issue (see various jenkins job results in the comment history) | 16:22 |
| *** pcrews has quit IRC | 16:22 | |
| malini | fungi: I am really confused on tht one as well | 16:24 |
| malini | I checked the repo to make sure that the ironic stuff is already there | 16:24 |
| malini | let me get you the repo lines | 16:24 |
| *** katyafervent_awa is now known as katyafervent | 16:25 | |
| devananda | fungi: http://logs.openstack.org/85/70885/1/check/check-tempest-dsvm-ironic-nv/170477a/logs/screen-n-cpu.txt.gz?level=AUDIT | 16:25 |
| devananda | fungi: looks like n-cpu didn't even DO much | 16:25 |
| devananda | fungi: i've been digging through the screen logs and dont see any errors, except the omission of nova doign any work | 16:26 |
| *** emagana has quit IRC | 16:26 | |
| *** miqui has quit IRC | 16:29 | |
| jnoller | fungi: did we ever see performance results? | 16:30 |
| *** emagana has joined #openstack-infra | 16:31 | |
| fungi | jnoller: i haven't. i think we need someone to put together a proper statistical analysis. also i noticed yesterday that over the weekend e stopped being able to build new images from that base image... haven't had time to dig into the logs yet and find out why | 16:32 |
| *** max_lobur has quit IRC | 16:33 | |
| malini | fungi: the only possibility I can think of is the ironic stuff was there when I originally submitted the patchset & got removed from the repo since then..See an older patch here https://review.openstack.org/#/c/65140/3/modules/openstack_project/files/zuul/layout.yaml | 16:33 |
| *** miqui has joined #openstack-infra | 16:33 | |
| malini | fungi: does tht imply rebase doesn't remove stuff ? or I am doing something weird | 16:34 |
| *** matrohon has joined #openstack-infra | 16:34 | |
| *** bookwar has quit IRC | 16:35 | |
| *** max_lobur has joined #openstack-infra | 16:35 | |
| fungi | malini: it means at some point when you rebased you probably misinterpreted one of the conflict resolution chunks and left those lines in when the rebase said they should no longer be in master | 16:35 |
| *** masayukig has joined #openstack-infra | 16:35 | |
| fungi | malini: you could just amend your patch to take those lines out of it and will probably be fine | 16:35 |
| malini | fungi: tht is very much possible..Checking now | 16:35 |
| sdague | jnoller: if you pull the latest - https://github.com/sdague/tempest-timings and run scrape.py you'll get the latest results. My understanding is check-rax-iad is the new image | 16:36 |
| sdague | that also calculates mean and std deviation timings on all the clouds | 16:37 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 16:37 |
| sdague | jnoller, fungi: http://dague.net/testing/check-tempest-dsvm-full.png and http://dague.net/testing/check-tempest-dsvm-postgres-full.png | 16:38 |
| *** thuc has joined #openstack-infra | 16:39 | |
| *** masayukig has quit IRC | 16:39 | |
| *** thuc_ has joined #openstack-infra | 16:40 | |
| fungi | sdague: jnoller: lower mean by about 15-20% and std stayed about the same i guess | 16:40 |
| *** jcooley_ has quit IRC | 16:40 | |
| sdague | fungi: yeh, that's my take | 16:41 |
| *** emagana has quit IRC | 16:41 | |
| *** markwash has joined #openstack-infra | 16:41 | |
| fungi | so it doesn't seem to address the variability in runtimes we're seeing | 16:41 |
| sdague | fungi: correct | 16:41 |
| jnoller | Interesting. | 16:41 |
| fungi | though it is a bit faster, at least | 16:41 |
| jnoller | You're using perf1-1 images, right? | 16:42 |
| sdague | that level of variability in rackspace cloud has been there forever though | 16:42 |
| jnoller | oh wait, 8gb | 16:42 |
| sdague | it's not new | 16:42 |
| *** markmcclain has joined #openstack-infra | 16:42 | |
| sdague | I think my first version of this scraper was over a year ago, and the results at the time were basically the same. | 16:43 |
| jnoller | there's too much variability for my taste. | 16:43 |
| jnoller | I think I need to talk to pvo and gabe | 16:43 |
| *** thuc has quit IRC | 16:43 | |
| markmcclain | sdague: have a bit of time to talk about lingering config drive/metadata service/key injection issues? | 16:43 |
| jnoller | need to think of a good summary though | 16:44 |
| sdague | jnoller: so the interesting thing about the variability, if you look at the graphs, is it's not continuous | 16:44 |
| jnoller | yeah | 16:44 |
| sdague | it's sort of clusters around a few different values | 16:44 |
| sdague | which I haven't done anything to pull those out with real math yet | 16:44 |
| jnoller | westmaas_ ^^ see above when you're back | 16:44 |
| jeblair | sdague: i assume hpcloud-b has very little data (we run _very_ few jobs on that, each graph probably has 2 points) so the std for that probably isn't supportable, but is the mean close enough to az1-3 to expand the region-b quota to get more data? | 16:44 |
| sdague | but my expectation is that what's really going on is we're exposing some non homogeneous part of the rax infrastructure, and we can actually see that difference in the bulk data | 16:45 |
| sdague | jeblair: sure, or just ignore region-b | 16:46 |
| westmaas_ | faster average, still variable? | 16:46 |
| westmaas_ | and this is just at runtime right? cause you pre-build instances? | 16:46 |
| jnoller | westmaas_: see http://dague.net/testing/check-tempest-dsvm-full.png http://dague.net/testing/check-tempest-dsvm-postgres-full.png | 16:46 |
| jeblair | sdague: ignore? that's the opposite of use more... what do you mean? | 16:46 |
| fungi | jeblair: we increased the number of jobs we're running on hpcloud-region-b to be the same as for the rax-iad check nodes... right now we have a min 1 with equal weighting between the two providers (to avoid having more than one rax-iad provider configured) | 16:46 |
| sdague | westmaas_: yeh, graphs above, as well updated scraper script | 16:46 |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 16:46 | |
| sdague | jeblair: I'm saying, don't care about the region-b data | 16:46 |
| * fungi gets some quick node counts | 16:47 | |
| sdague | realistically, we could collect more data on it | 16:47 |
| jeblair | sdague: oh, well, it's the future and az1-3 are deprecated so we should probably be somewhat interested in it. :) | 16:47 |
| sdague | but my eyeball over time is that the hp azs are pretty equivalent | 16:47 |
| sdague | ok, so sure, run more on it to see how it reacts | 16:47 |
| westmaas_ | we have some different manufacturers in different DCs...is that that straight line the HP region b or the rax ord? | 16:47 |
| jnoller | westmaas_: the black line is perf 8gb in IAD | 16:48 |
| westmaas_ | or too suspiciously straight | 16:48 |
| devananda | fungi: i see the failure that seems to be hitting our gate affecting others too | 16:48 |
| fungi | jeblair: oh, nevermind. the max for hpcloud-region-b keeps us from adding more than one node at a time | 16:48 |
| sdague | westmaas_: so realize it's a line graph... so less than idea. I'd look at the mean / std for most of the bits | 16:48 |
| westmaas_ | jnoller: yeah we have both manufacturers in iad :) | 16:48 |
| devananda | fungi: logstash search for: virSecurityDACRestoreSecurityFileLabel AND build_status:"FAILURE" | 16:48 |
| jeblair | westmaas_: i'm assuming region b is the straight line due to the low std | 16:48 |
| sdague | jeblair: yeh, it's probably 2 data points total | 16:48 |
| westmaas_ | sdague: yeah, k | 16:48 |
| sdague | so will always be a straight line | 16:49 |
| devananda | fungi: actually, ignore that | 16:49 |
| devananda | red herring | 16:49 |
| *** bhuvan has joined #openstack-infra | 16:49 | |
| westmaas_ | ok - I'm dealing with some other issues today, but will circle back on this with our perf testers | 16:49 |
| sdague | westmaas_: https://github.com/sdague/tempest-timings - ./scrape.py -s will dump the timing data out to json as well | 16:49 |
| westmaas_ | thanks for the help guys, really appreciate it | 16:49 |
| westmaas_ | we'll get this better | 16:49 |
| sdague | so you can look at actually number instead of silly graphs | 16:50 |
| *** pcrews has joined #openstack-infra | 16:50 | |
| *** marun has joined #openstack-infra | 16:50 | |
| sdague | markmcclain: so honestly, I'm not entirely current on the issue, and need to actually try to get back to sleep (it's a very late/early local time in .kr right now) :) | 16:51 |
| *** mestery has joined #openstack-infra | 16:51 | |
| *** mestery has quit IRC | 16:51 | |
| *** AlexF has joined #openstack-infra | 16:51 | |
| *** zehicle_at_dell has joined #openstack-infra | 16:52 | |
| *** dprince has quit IRC | 16:52 | |
| *** bookwar has joined #openstack-infra | 16:52 | |
| *** rcleere has joined #openstack-infra | 16:52 | |
| *** mestery has joined #openstack-infra | 16:53 | |
| markmcclain | sdague: ok.. you can ping me when you're in awake hours (in PST this week) | 16:53 |
| *** CaptTofu has quit IRC | 16:53 | |
| markmcclain | *I'm PST this wk | 16:53 |
| *** thedodd has joined #openstack-infra | 16:54 | |
| *** Ryan_Lane has joined #openstack-infra | 16:55 | |
| *** afazekas has quit IRC | 16:55 | |
| zaro | morning | 16:55 |
| *** smurugesan has joined #openstack-infra | 16:56 | |
| *** boris-42_ has joined #openstack-infra | 16:57 | |
| devananda | fungi: ok - really fouind it this time. n-cpu seems to just stop without any log message, then all subsequent attempts to schedule fail with "Filter ComputeFilter returned 0 hosts" | 16:58 |
| zaro | pleia2: is it bugaday? | 16:58 |
| devananda | fungi: logstask search for: build_status:"FAILURE" AND filename:"logs/screen-n-sch.txt" AND "Filter ComputeFilter returned 0 hosts" | 16:59 |
| *** zehicle_at_dell has quit IRC | 16:59 | |
| fungi | mordred: i opened https://launchpad.net/bugs/1276242 just now--you might be interested... it's pypi-mirror/wheel related | 16:59 |
| *** zehicle_at_dell has joined #openstack-infra | 16:59 | |
| *** e0ne_ has quit IRC | 16:59 | |
| pleia2 | zaro: it is! starting now :) | 17:00 |
| pleia2 | happy bug day | 17:00 |
| pleia2 | etherpad here: https://etherpad.openstack.org/p/cibugreview-february2014 | 17:00 |
| *** zehicle_at_dell has quit IRC | 17:00 | |
| devananda | fungi: can we push https://review.openstack.org/#/c/70544/ through now? | 17:00 |
| *** AlexF has quit IRC | 17:00 | |
| *** zehicle_at_dell has joined #openstack-infra | 17:00 | |
| *** boris-42_ is now known as boris-42 | 17:00 | |
| zaro | fungi: you think you can do quick approval here to unblock my testing? https://review.openstack.org/70865 | 17:00 |
| devananda | fungi: this is, afaict, the only substantial difference between ironic's d-g and nova's, and our gate has been wedged since friday night ... | 17:01 |
| *** CaptTofu has joined #openstack-infra | 17:01 | |
| devananda | clarkb: or if you're around ^ | 17:01 |
| fungi | devananda: i have no problem with it... it's mostly between ironic and qa. i can review it on its correctness as a job configuration patch | 17:02 |
| *** jcoufal_ has joined #openstack-infra | 17:02 | |
| *** jcoufal has quit IRC | 17:02 | |
| jeblair | zaro: when you have a minute, can you look into combining the WIP and Approval functions into one column? | 17:02 |
| jeblair | zaro: so instead of having a "WIP" column and an "Approved" column, you have a "Workflow" column with -1=WIP and +1=Approved... | 17:02 |
| *** mestery has quit IRC | 17:02 | |
| *** dizquierdo has quit IRC | 17:03 | |
| *** rockstar has joined #openstack-infra | 17:03 | |
| *** rockstar has joined #openstack-infra | 17:03 | |
| jeblair | zaro: (with the usual "at least one +1 and also no -1 votes" requirement) | 17:03 |
| *** mestery has joined #openstack-infra | 17:03 | |
| *** boris-42 is now known as boris-42_ | 17:04 | |
| *** mestery has quit IRC | 17:05 | |
| *** markmc has quit IRC | 17:05 | |
| *** mestery has joined #openstack-infra | 17:06 | |
| zaro | jeblair: ok. i'll take a look. | 17:06 |
| *** hcc is now known as hdd | 17:07 | |
| reed | jeblair, fungi: smarcet fixed the comments made by Marton, please check https://review.openstack.org/#/c/69636/ | 17:08 |
| *** slong has joined #openstack-infra | 17:09 | |
| *** gyee has joined #openstack-infra | 17:10 | |
| *** mestery_ has joined #openstack-infra | 17:10 | |
| *** gyee has quit IRC | 17:11 | |
| *** mestery_ has quit IRC | 17:11 | |
| *** mestery has quit IRC | 17:11 | |
| *** mestery has joined #openstack-infra | 17:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8 https://review.openstack.org/70865 | 17:12 |
| *** gyee has joined #openstack-infra | 17:13 | |
| *** sarob has joined #openstack-infra | 17:13 | |
| *** rakhmerov has quit IRC | 17:15 | |
| pleia2 | jeblair, clarkb - bug day :) https://etherpad.openstack.org/p/cibugreview-february2014 | 17:16 |
| *** sarob has quit IRC | 17:20 | |
| zaro | fungi: thank for quick aprv. | 17:21 |
| *** AlexF has joined #openstack-infra | 17:21 | |
| *** talluri has joined #openstack-infra | 17:23 | |
| *** salv-orlando has quit IRC | 17:23 | |
| devananda | jeblair: have a moment to help me try to unwedge Ironic's gate? https://review.openstack.org/#/c/70544/2 needs a +A | 17:25 |
| *** jasondotstar has quit IRC | 17:25 | |
| *** salv-orlando has joined #openstack-infra | 17:26 | |
| *** pcrews has quit IRC | 17:26 | |
| *** oubiwann_ has quit IRC | 17:28 | |
| lifeless | fungi: it's not, because you had it offline I didn't kill myself getting it up before bed | 17:29 |
| lifeless | fungi: I will be back on it after my morning-of-hell-meetings | 17:29 |
| fungi | lifeless: okay--we didn't want to add it back to the nodepool config until it was up and running again, so stalemate on that last night i guess | 17:30 |
| *** AlexF has quit IRC | 17:31 | |
| *** mestery_ has joined #openstack-infra | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| *** amotoki has quit IRC | 17:34 | |
| *** jcooley_ has joined #openstack-infra | 17:34 | |
| jamesmcarthur | Hey guys. Just a heads up that we are about to do a deployment on OpenStack.org. Any services/graphics that are in use from this domain may see a temporary outage. We're expecting about 5-10 minutes of down time during this maintenance window. | 17:35 |
| fungi | jamesmcarthur: thanks! i saw toddmorey's e-mail yesterday too. good luck! | 17:36 |
| *** masayukig has joined #openstack-infra | 17:36 | |
| *** rakhmerov has joined #openstack-infra | 17:36 | |
| *** AlexF has joined #openstack-infra | 17:36 | |
| jamesmcarthur | thank you sir! | 17:37 |
| jeblair | pleia2: ack | 17:37 |
| *** mestery has quit IRC | 17:38 | |
| *** moted has joined #openstack-infra | 17:38 | |
| *** jamesmcarthur has left #openstack-infra | 17:38 | |
| *** masayukig has quit IRC | 17:41 | |
| *** sarob has joined #openstack-infra | 17:41 | |
| *** sandywalsh_ has quit IRC | 17:42 | |
| *** sandywalsh has joined #openstack-infra | 17:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make Ironic tests stop using file injection https://review.openstack.org/70544 | 17:43 |
| *** SpamapS_ is now known as SpamapS | 17:43 | |
| *** jasondotstar has joined #openstack-infra | 17:44 | |
| *** jcooley_ has quit IRC | 17:44 | |
| jeblair | grr, i thought we didn't link to openstack.org for any of the css, etc for status.o.o | 17:44 |
| *** AlexF has quit IRC | 17:44 | |
| *** Ajaeger has joined #openstack-infra | 17:45 | |
| fungi | i thought we decided we shouldn't, but don't recall whether it was fixed | 17:45 |
| *** fbo has quit IRC | 17:46 | |
| *** msuriar has quit IRC | 17:46 | |
| *** cp16net has quit IRC | 17:46 | |
| pleia2 | anteaya: is there work that can be done without Hunner's puppet board configs for apache? I'm happy to keep working on this, I just thought we were waiting on him | 17:46 |
| *** sarob has quit IRC | 17:46 | |
| pleia2 | (not wanting to reinvent the wheel and all) | 17:46 |
| anteaya | I don't know what to do without him, pleia2 | 17:47 |
| fungi | yep, i cound at least 5 css files linked from www.o.o in the status.o.o page | 17:47 |
| anteaya | he had set up the example locally and stalled on getting it into our config | 17:47 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Reduce rax-iad usage by 50% https://review.openstack.org/71042 | 17:47 |
| anteaya | on the db issue, if I remember correctly | 17:47 |
| anteaya | since it requires puppetdb and we don't have that in our config | 17:47 |
| jeblair | fungi: pls spdy aprv https://review.openstack.org/71042 ? | 17:47 |
| jeblair | antonym: https://review.openstack.org/71042 | 17:48 |
| pleia2 | anteaya: ok, I updated your comment in the etherpad, it's not really stalled on us ;) | 17:48 |
| anteaya | last I talked to him, he was going to talk to his manager to get some time for this | 17:48 |
| pleia2 | ok, thanks | 17:48 |
| anteaya | pleia2: thank you, yes not stalled on us, sorry my comment left that impression | 17:48 |
| openstackgerrit | mark mcclain proposed a change to openstack-infra/config: Enable config drive on all Neutron jobs https://review.openstack.org/71043 | 17:48 |
| *** cp16net has joined #openstack-infra | 17:48 | |
| *** burt1 has joined #openstack-infra | 17:49 | |
| *** msuriar has joined #openstack-infra | 17:49 | |
| *** sarob has joined #openstack-infra | 17:49 | |
| devananda | markmcclain: heh :) | 17:50 |
| fungi | jeblair: i'll hand-apply 71042 on nodepool.o.o now, since puppet is intentionally off there while the tripleo-ci cloud continues its restoration | 17:50 |
| jeblair | fungi: ok, thx | 17:50 |
| markmcclain | devananda: yeah I'd like to get patches merging again | 17:51 |
| *** AlexF has joined #openstack-infra | 17:51 | |
| devananda | markmcclain: nearly the same patch for ironic's tests just went through, i'm watching to see if that fixes it | 17:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Reduce rax-iad usage by 50% https://review.openstack.org/71042 | 17:52 |
| antonym | jeblair: thanks! | 17:52 |
| fungi | updated | 17:53 |
| *** amcrn has joined #openstack-infra | 17:53 | |
| Ajaeger | Hi infra team, looking at one of the bugs on etherpad, I can tackle it if I get some advise: https://bugs.launchpad.net/openstack-ci/+bug/1216153 . The job is at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/manuals.yaml#n111 - and I have one question: Tom proposes to use generatedocbook from openstack-doc-tools instead of calling ./tools/generatedocbook directly. H | 17:53 |
| *** esker has quit IRC | 17:54 | |
| anteaya | pleia2: though I will say in fairness to Hunner that I haven't been around to talk to him about it | 17:54 |
| *** salv-orlando_ has joined #openstack-infra | 17:54 | |
| *** esker has joined #openstack-infra | 17:54 | |
| *** dprince has joined #openstack-infra | 17:54 | |
| jeblair | Ajaeger: you were cut off at "./tools/generatedocbook directly. H" | 17:54 |
| *** gokrokve_ has quit IRC | 17:54 | |
| Ajaeger | jeblair: oh, let me repaste again | 17:55 |
| *** fbo has joined #openstack-infra | 17:55 | |
| Ajaeger | Tom proposes to use generatedocbook from openstack-doc-tools instead of calling ./tools/generatedocbook directly. How can we do this best? | 17:55 |
| jeblair | Ajaeger: irc has a 512 char limit iirc | 17:55 |
| *** oubiwann_ has joined #openstack-infra | 17:55 | |
| Ajaeger | jeblair: Sorry, didn't count ;) | 17:55 |
| *** resker has joined #openstack-infra | 17:55 | |
| *** gokrokve_ has joined #openstack-infra | 17:55 | |
| fungi | judging from the test nodes graph, nodepool doesn't seem to be deleting nodes as quickly following the merger of the aggressive deletes patch series last night | 17:56 |
| fungi | clarkb: ^ | 17:56 |
| *** salv-orlando_ has quit IRC | 17:56 | |
| *** esker has quit IRC | 17:57 | |
| fungi | i was mostly paying attention to things which slipped past the 3 hour mark without being culled, but there's a bunch between 2 and 3 hours in delete | 17:57 |
| *** salv-orlando_ has joined #openstack-infra | 17:57 | |
| jeblair | Ajaeger: i think the best plan would be: change that to a freestyle project; use virtualenv (possibly via tox) to pip install openstack-doc-tools; run the tool; then "mvn whatever..." | 17:57 |
| *** salv-orlando has quit IRC | 17:57 | |
| *** salv-orlando_ is now known as salv-orlando | 17:57 | |
| *** Guest54041 has joined #openstack-infra | 17:58 | |
| jeblair | Ajaeger: (strictly speaking, you could probably do that without changing the maven aspect, and just make a pre-builder that did the virtualenv bit) | 17:58 |
| *** Guest54041 has quit IRC | 17:58 | |
| jeblair | Ajaeger: (but we are going to need to stop using maven jobs in jenkins soon anyway, so i'm trying to trick you into doing that at the same time while you're working on the job) :) | 17:58 |
| *** harlowja_away is now known as harlowja | 17:59 | |
| *** elasticio has joined #openstack-infra | 17:59 | |
| mordred | ++ | 17:59 |
| * mordred likes when jeblair is tricky | 17:59 | |
| *** elasticio_ has joined #openstack-infra | 17:59 | |
| jeblair | mordred: the best part is he'll never suspect ;) | 17:59 |
| *** elasticio is now known as Guest7575 | 17:59 | |
| mordred | jeblair: nice one! | 17:59 |
| *** Guest7575 has quit IRC | 17:59 | |
| fungi | tricksy jeblair. we hates maven, we does | 17:59 |
| *** elasticio_ is now known as Guest58289 | 17:59 | |
| *** Guest58289 has quit IRC | 17:59 | |
| anteaya | markmcclain: before I vote on 71043 do we have any proof this will work? or just revert if it doesn't? | 17:59 |
| *** AlexF has quit IRC | 17:59 | |
| Ajaeger | jeblair: If you want me to convert all maven jobs, just tell me the whole story - the openstack-doc-test tool calls maven directly already. We can replace the maven stuff with jobs that publish. | 17:59 |
| Ajaeger | jeblair: have a look at https://review.openstack.org/#/c/70594/. | 18:00 |
| Ajaeger | I can change that doc-publish job to build *all* books in one go and we then publish from publish-docs dir. | 18:00 |
| *** KurtMartin is now known as kmartin | 18:00 | |
| *** 65MAAIZIK has joined #openstack-infra | 18:00 | |
| *** elasticio__ has joined #openstack-infra | 18:00 | |
| Ajaeger | the checkbuild tox test calls maven already... | 18:00 |
| *** elasticio__ is now known as Guest5187 | 18:01 | |
| Ajaeger | Is that a direction we should go? | 18:01 |
| *** morganfainberg_Z is now known as morganfainberg | 18:01 | |
| Ajaeger | In that case, I could change the translation job as a first test... | 18:01 |
| *** hashar has quit IRC | 18:01 | |
| jeblair | Ajaeger: awesome; yes -- we need to get rid of maven-based jenkins jobs because we want to get rid of jenkins (calling maven directly from shell is obviously fine, we don't care about that) | 18:02 |
| *** Guest5187 has quit IRC | 18:02 | |
| markmcclain | anteaya: when looking at https://review.openstack.org/#/c/67537/ | 18:02 |
| *** 65MAAIZIK has quit IRC | 18:02 | |
| markmcclain | you'll notice that the tests with config_drive enabled pass | 18:02 |
| markmcclain | and do not trigger the nova-compute bug which the nova team is currently working on | 18:02 |
| * anteaya clicks to notice | 18:02 | |
| Ajaeger | jeblair: can you review 70594 and tell me whether that logic for publishing is fine, please? Based on the review for that, I handle bug 1216153 ... and then do the other maven jobs... | 18:03 |
| Ajaeger | (the Documentation team related ones only) | 18:03 |
| jeblair | Ajaeger: sure; i'm just back from a bit of traveling, so i have some catching up to do, but i starred that one and will try to look at it early on in my reviewing to keep you from being blocked too long | 18:04 |
| clarkb | morning | 18:04 |
| clarkb | fungi: as quickly compared to before the merge? | 18:04 |
| *** nati_ueno has joined #openstack-infra | 18:04 | |
| mordred | jeblair: I need to run out - I may not be able to make it back by the infra meeting - but I'm going to try my best | 18:04 |
| fungi | clarkb: far, far less quickly | 18:04 |
| clarkb | fungi: ah yes, I see the vast regions of purple | 18:05 |
| *** max_lobur is now known as max_lobur_afk | 18:05 | |
| *** apevec has joined #openstack-infra | 18:05 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov | 18:05 | |
| Ajaeger | jeblair: ok. | 18:05 |
| jeblair | pleia2, clarkb, fungi: i will try to work on triaging | 18:06 |
| fungi | jeblair: i've started triaging as well | 18:06 |
| fungi | clarkb: do you think 67986 is likely to help? | 18:06 |
| jeblair | fungi: ok. are you starting from the top or bottom? :) | 18:06 |
| apevec | hi, I have grenade g->h issue: http://logs.openstack.org/49/66149/2/check/check-grenade-dsvm/50595f9/logs/old/screen-c-vol.txt.gz | 18:06 |
| fungi | clarkb: i haven't looked at the logs yet to see whether it's throwing exceptions when deleting | 18:07 |
| clarkb | fungi: it may | 18:07 |
| fungi | jeblair: at the top, but i can switch | 18:07 |
| apevec | "old" Grizzly fails with pkg_resources.DistributionNotFound: SQLAlchemy>=0.7.3,<=0.7.9 | 18:07 |
| jeblair | fungi: no prob, i'll start at bottom | 18:07 |
| clarkb | since I believe the current deletion is do a delete not blocking, if that fails then we wait | 18:07 |
| apevec | pip freeze shows SQLAlchemy==0.7.10 | 18:07 |
| clarkb | fungi: I can approve that change and see if it helps | 18:07 |
| reed | thanks fungi | 18:07 |
| * reed away european time | 18:07 | |
| fungi | clarkb: well, we'd need to hand-patch it onto the nodepool server | 18:07 |
| apevec | sdague, ^^^ are old and new sharing the same venv in grande? | 18:07 |
| *** slong has quit IRC | 18:07 | |
| openstackgerrit | Philippe Godin proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin. https://review.openstack.org/69608 | 18:08 |
| clarkb | fungi: oh right, I will approve and hand patch then | 18:08 |
| *** e0ne has joined #openstack-infra | 18:08 | |
| clarkb | then swing around and do bugs with everony else | 18:08 |
| fungi | clarkb: tripleo cloud is still offline, but lifeless hopes to finish restoring it to service after his day-o-meetins | 18:08 |
| anteaya | markmcclain: okay let's see what this does, +1'ing | 18:09 |
| clarkb | fungi: done | 18:09 |
| markmcclain | anteaya: dansmith may have fixed the bug on the nova side | 18:09 |
| markmcclain | waiting for the experimental checks to complete on his patch | 18:09 |
| anteaya | yay dansmith | 18:09 |
| anteaya | patch #? | 18:09 |
| markmcclain | https://review.openstack.org/71046 | 18:10 |
| anteaya | thanks | 18:10 |
| clarkb | devananda: fungi: I thought lifeless actually fixed devstack | 18:10 |
| lifeless | clarkb: I did | 18:10 |
| lifeless | and dansmith has just fixed nova (pending confirmation by tempest) | 18:11 |
| clarkb | great, then we shouldn't need to change the ironic jobs (unless something else cascaded into broken) | 18:11 |
| dansmith | lifeless: hold on there, cowboy :) | 18:11 |
| lifeless | see 71046,1 | 18:11 |
| dansmith | lifeless: dansmith has just proposed a hopeful fix :) | 18:11 |
| lifeless | clarkb: fixing ironic is fairly important; I'm not suggesting not merging deva's thing | 18:11 |
| clarkb | lifeless: I am | 18:11 |
| clarkb | lifeless: if the thing is fixed why would we 'fix' it somewhere else? | 18:12 |
| lifeless | dansmith: I did say pending confirmation :) | 18:12 |
| lifeless | clarkb: because as I said for devstack fixing the thing in front of us might not fix ironic | 18:12 |
| dansmith | lifeless: it wasn't enough speculative-sounding for me :D | 18:12 |
| lifeless | clarkb: we don't know how deep the rabbit hole goes | 18:12 |
| clarkb | but we won't know either unless we test it | 18:13 |
| lifeless | clarkb: but we do know that Ironic doesn't care about any of it for now, it just wants neutron running for it's api tests until it gets it's own nova driver in the mix | 18:13 |
| clarkb | which devananda's change prevents | 18:13 |
| devananda | lifeless: the hopeful fix for ironic (stop disabling configdrive) has merged | 18:13 |
| lifeless | devananda: cool | 18:13 |
| lifeless | clarkb: I thought deva's change was in ironic context only? | 18:13 |
| devananda | also, fwiw, we arent starting up neutron in ironic's d-g env at the moment | 18:13 |
| devananda | lifeless: it should be only in ironic tests, yes | 18:14 |
| clarkb | lifeless: it is but as you said yesterday we need to test it and neutron and neutron isolated | 18:14 |
| clarkb | as there may be other problems | 18:14 |
| lifeless | clarkb: yes, but they are separate concerns - no need to hold ironic's jobs hostage | 18:14 |
| *** sarob has quit IRC | 18:14 | |
| lifeless | clarkb: my concern there was only ever that I thought config drive on forced metadata off, which it does not. | 18:14 |
| *** sarob has joined #openstack-infra | 18:15 | |
| lifeless | it just means that vm's don't *talk* to metadata because they have config drive locally. | 18:15 |
| lifeless | clarkb: and that matters in the holistic what-are-we-testing discussion | 18:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Cleanup old servers every minute. https://review.openstack.org/67986 | 18:15 |
| lifeless | clarkb: but ironic doesn't count *at all* in that space, yet, not for nova. | 18:15 |
| Ajaeger | clarkb: I need to pass an extra parameter to tox. So, instead of adding lots of "build-lang-$LANG" entries to tox.ini and calling these from run-tox.sh, I'd like to run "tox -e build-lang -- $LANG". Is that directly possible? | 18:15 |
| *** coolsvap is now known as coolsvap_away | 18:16 | |
| jeblair | Ajaeger: you can pass the extra command line args given to tox on to the program it runs; there should be some examples in the python tox.ini files | 18:17 |
| jeblair | Ajaeger: the 'venv' env comes to mind | 18:18 |
| *** talluri has quit IRC | 18:18 | |
| clarkb | and if you need that in the jenkins jobs we can probalby have run-unittests do a shift then pass $@ to tox | 18:18 |
| clarkb | lifeless: I understand but yesterday you were arguing for testing that the fix fixed everything, we will now not actually test that | 18:19 |
| lifeless | clarkb: I was arguing for making sure it fixed things before we merged it yes | 18:19 |
| lifeless | clarkb: I didn't want to run of saying 'all good, next in queue!' if that wasn't the case | 18:20 |
| *** nicedice has joined #openstack-infra | 18:20 | |
| lifeless | clarkb: it's still entirely possible ironic will have other issues; which will lurk for a while :( but deva needs to land stuff - cons and pros. | 18:20 |
| *** oubiwann_ has quit IRC | 18:22 | |
| *** AlexF has joined #openstack-infra | 18:22 | |
| *** oubiwann_ has joined #openstack-infra | 18:22 | |
| clarkb | fungi: jeblair: pleia2: where should I jump on bugs? | 18:23 |
| *** pcrews has joined #openstack-infra | 18:23 | |
| jeblair | clarkb: fungi and i are triaging new ones; you want to scan the list and look for any we could close or drop? | 18:24 |
| clarkb | fungi: delete state nodes fell from 281 to 227 over the last few minutes | 18:24 |
| *** jcooley_ has joined #openstack-infra | 18:24 | |
| clarkb | jeblair: yup | 18:24 |
| devananda | clarkb: roughly how long before the merge to https://review.openstack.org/#/c/70544/ trickles down into test runs? ~ an hour, ya? | 18:24 |
| clarkb | devananda: 10-15 minutes after the */15 cron | 18:25 |
| devananda | k | 18:25 |
| fungi | clarkb: jeblair and i are going through new bugs from both ends, so you could either start grabbing some out of the middle or dig into the ones in teh etherpad you know merit status updates | 18:25 |
| fungi | clarkb: excellent | 18:25 |
| clarkb | mordred: want to add anything to https://bugs.launchpad.net/openstack-ci/+bug/1242569 ? | 18:25 |
| clarkb | fungi has it up to date, not sure if there are additional things you want to add | 18:26 |
| *** jcooley_ has quit IRC | 18:26 | |
| fungi | is the log error checker script in infra/config as a slave script, or in devstack, tempest, elsewhere? | 18:26 |
| clarkb | fungi: tempest | 18:27 |
| clarkb | under tools/ iirc | 18:27 |
| fungi | clarkb: thanks (one more bug i can move to another project) | 18:27 |
| *** dkehn has quit IRC | 18:27 | |
| clarkb | fungi: for https://bugs.launchpad.net/openstack-ci/+bug/1010621, once I backup eavesdrop any chance you want to do some of the puppetmaster work? | 18:27 |
| fungi | clarkb: sure | 18:28 |
| clarkb | fungi: great (I should hurry up and do eavesdrop, it should be easy) | 18:28 |
| *** dkehn has joined #openstack-infra | 18:28 | |
| *** thuc_ has quit IRC | 18:28 | |
| *** thuc has joined #openstack-infra | 18:29 | |
| clarkb | fungi: zaro: for https://bugs.launchpad.net/openstack-ci/+bug/1267364 I feel like we should take the time to report that upstream since upstream claims to have fixed it but it clearly isn't fixed | 18:30 |
| clarkb | at the very least poke the bug that indicated 1.542 had the fix and say 1.543 is not fixed | 18:30 |
| *** jooools has quit IRC | 18:30 | |
| fungi | clarkb: unfortunately since that report i've not caught another failure on a slave assigned to one of the upgraded jenkins masters quickly enough to catch the console log before it gets expired | 18:31 |
| *** wendar_ is now known as wendar | 18:31 | |
| clarkb | fungi: I bet having single use slaves makes it hard | 18:31 |
| clarkb | I wonder if those tracebacks show up in the jenkins /var/log/ log somewhere | 18:32 |
| fungi | also since adding more masters and getting a handle on the slave churn, the count of reports has dropped off (i don't even know whether we're still seeing that particular traceback any longer) | 18:32 |
| *** e0ne_ has joined #openstack-infra | 18:33 | |
| *** thuc has quit IRC | 18:33 | |
| *** changbl has joined #openstack-infra | 18:34 | |
| clarkb | fungi: down to 86 slaves in the delete state | 18:34 |
| fungi | clarkb: though the ready node count looks to have jumped up | 18:34 |
| *** e0ne has quit IRC | 18:35 | |
| *** gokrokve_ has quit IRC | 18:35 | |
| *** gokrokve has joined #openstack-infra | 18:36 | |
| *** thuc has joined #openstack-infra | 18:36 | |
| *** masayukig has joined #openstack-infra | 18:36 | |
| *** che-arne has quit IRC | 18:37 | |
| *** mestery_ has quit IRC | 18:38 | |
| *** dprince has quit IRC | 18:39 | |
| *** yassine has quit IRC | 18:39 | |
| clarkb | pleia2: https://bugs.launchpad.net/openstack-ci/+bug/1082795 any chance you can upgrade that with the state of latest and greatest tripleo testing | 18:39 |
| zaro | fungi, clarkb : so could that be that we had too many connected slaves to the master? | 18:39 |
| *** jpich has quit IRC | 18:40 | |
| clarkb | jeblair: https://review.openstack.org/#/c/69442/ fixes a 'high' bug. Didn't note that in the commit message because I didn't know there was a preexisting high bug for the problem. I can update the commit message if you like, but that should probably get eyes on it if possible | 18:40 |
| fungi | zaro: well, we've seen various slave agent communication failures in the past when we scaled up either the number of slaves or possibly rate at which we were adding and removing them | 18:40 |
| clarkb | zaro: sure, but its still a bug in jenkins | 18:40 |
| *** markmcclain has quit IRC | 18:40 | |
| pleia2 | clarkb: sure thing | 18:41 |
| *** masayukig has quit IRC | 18:41 | |
| *** gokrokve has quit IRC | 18:41 | |
| clarkb | 'too many slaves' is not something we should be saying about a jenkins with 100 slaves | 18:41 |
| *** shivh has joined #openstack-infra | 18:41 | |
| *** sdake has quit IRC | 18:42 | |
| clarkb | fungi: re six not getting latest versions installed, do we know what is happening there? is mirror build failing before it gets to six? | 18:42 |
| *** sdake has joined #openstack-infra | 18:43 | |
| *** sdake has quit IRC | 18:43 | |
| *** sdake has joined #openstack-infra | 18:43 | |
| fungi | clarkb: that's my suspicion. i only found time to document the symptom, but not yet time to start pouring over the logs | 18:44 |
| jeblair | zaro, clarkb, fungi: i don't believe the noclassdeffound error is correctible | 18:44 |
| fungi | i've got it up in a tab and will try to unearth more details if nobody beats me to it | 18:44 |
| *** dims has joined #openstack-infra | 18:44 | |
| clarkb | jeblair: at the very least jenkins should mark the node offline and reschedule the job that would have failed on a different node | 18:45 |
| jeblair | zaro, clarkb, fungi: i mean, it may be worth mentioning on the upstream bug, but i think at this point we have found that if a job is aborted at just the right time, there is no way in java to correct that. | 18:45 |
| jeblair | clarkb: yeah, or perhaps try to detect it and re-cycle the node (offline-online) | 18:45 |
| clarkb | mordred: https://bugs.launchpad.net/openstack-ci/+bug/1254275 is that something that is still a problem? | 18:46 |
| jeblair | clarkb: i sort of doubt it would be a priority for them because most people don't abort jobs at scale. :) | 18:46 |
| fungi | worse, it doesn't offline the node at all, and just keeps spamming it with more jobs | 18:46 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 18:46 |
| Ajaeger | clarkb, jeblair: I'm calling tox now directly in the change above so that you see what I want to do. | 18:47 |
| Ajaeger | Your hints with run-unittests and venv didn't help me ;( | 18:48 |
| clarkb | jeblair: for https://bugs.launchpad.net/openstack-ci/+bug/1270319 you indicated it was a one line fix in geard, can I assign that bug to you? | 18:49 |
| jeblair | clarkb: sure | 18:49 |
| *** VijayT has joined #openstack-infra | 18:49 | |
| clarkb | done | 18:50 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 18:50 |
| clarkb | https://review.openstack.org/#/c/69969/ is another fix for a high bug | 18:50 |
| *** thedodd has quit IRC | 18:51 | |
| jeblair | i'm retargeting clouddocs-maven-plugin bugs to openstack-manuals since that tool is not part of openstack-infra | 18:51 |
| zaro | jeblair: i believe you have the power to do something about these? https://bugs.launchpad.net/openstack-ci/+bug/1247866 and https://bugs.launchpad.net/openstack-ci/+bug/1260983 | 18:52 |
| jeblair | Ajaeger: sorry; i'll try to be more helpful when i look more closely at your changes | 18:52 |
| clarkb | pleia2: https://bugs.launchpad.net/openstack-ci/+bug/995592 is that something I should assign to you? | 18:52 |
| clarkb | jeblair: ++ | 18:53 |
| Ajaeger | jeblair: no problem at all. Look closely and tell me what needs to be changed. I implement the tools support for this now ;) Thanks! | 18:53 |
| clarkb | zaro: pleia2 has been working on getting fedora to work with nodepool recently and making nodepool work with centos should be very similar. Once we have that we can kill the static slaves I think | 18:53 |
| pleia2 | clarkb: ah yes, I'll grab that one | 18:53 |
| clarkb | pleia2: thanks | 18:54 |
| *** jcooley_ has joined #openstack-infra | 18:55 | |
| *** freyes has joined #openstack-infra | 18:55 | |
| *** gokrokve has joined #openstack-infra | 18:55 | |
| *** andreaf has quit IRC | 18:55 | |
| *** sdake has quit IRC | 18:55 | |
| *** melwitt has joined #openstack-infra | 18:56 | |
| jeblair | zaro: scp-plugin pr merged | 18:57 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 18:58 |
| *** gokrokve has quit IRC | 19:00 | |
| *** markmcclain1 has joined #openstack-infra | 19:00 | |
| pleia2 | meeting time | 19:01 |
| *** markmcclain1 has quit IRC | 19:01 | |
| *** bhuvan has quit IRC | 19:02 | |
| *** bhuvan_ has joined #openstack-infra | 19:02 | |
| *** jnoller has quit IRC | 19:03 | |
| *** mestery has joined #openstack-infra | 19:04 | |
| *** jcoufal_ is now known as jcoufal | 19:06 | |
| *** markmcclain has joined #openstack-infra | 19:06 | |
| *** gokrokve has joined #openstack-infra | 19:06 | |
| *** dprince has joined #openstack-infra | 19:06 | |
| *** e0ne_ has quit IRC | 19:09 | |
| *** AlexF has quit IRC | 19:09 | |
| *** gokrokve has quit IRC | 19:11 | |
| *** rossella_s has quit IRC | 19:13 | |
| *** salv-orlando has quit IRC | 19:13 | |
| *** gokrokve has joined #openstack-infra | 19:14 | |
| *** salv-orlando has joined #openstack-infra | 19:14 | |
| *** jcooley_ has quit IRC | 19:17 | |
| *** reed has quit IRC | 19:18 | |
| *** reed has joined #openstack-infra | 19:19 | |
| *** sdake has joined #openstack-infra | 19:19 | |
| *** sdake has quit IRC | 19:19 | |
| *** sdake has joined #openstack-infra | 19:19 | |
| *** mriedem has quit IRC | 19:19 | |
| *** smarcet has left #openstack-infra | 19:20 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 19:22 |
| *** FallenPegasus has joined #openstack-infra | 19:23 | |
| jeblair | mikal: do you know if matthewoliver is/will be around? | 19:24 |
| SlickNik | dtroyer: around? | 19:26 |
| SlickNik | Had a questions for you. | 19:26 |
| SlickNik | question* | 19:26 |
| *** rakhmerov has quit IRC | 19:26 | |
| *** mriedem has joined #openstack-infra | 19:26 | |
| *** jasondotstar has quit IRC | 19:27 | |
| *** pblaho has joined #openstack-infra | 19:27 | |
| SlickNik | So to install swift via devstack, it's preferable to explicitly add the swift services to ENABLED_SERVICES, right? | 19:27 |
| dtroyer | SlickNik: yes | 19:28 |
| SlickNik | Today, the trove scripts add "ENABLED_SERVICES += swift" which seems to install all 3 swift services, but I don't think that's good practice. | 19:29 |
| SlickNik | especially if you consider multi-node devstack installs. | 19:29 |
| dtroyer | SlickNik: right…just been talking to juice and hub_cap about that… I think we're on the same page now | 19:29 |
| *** rakhmerov has joined #openstack-infra | 19:30 | |
| SlickNik | Cool, I'll make a change to the trove CI scripts to install the services explicitly. Much thanks for the confirmation! | 19:30 |
| SergeyLukjanov | devananda, clarkb, I'm writing about https://review.openstack.org/#/c/70001/ (Remove Neutron from Ironic tempest test pipelines) | 19:30 |
| juice | slicknik - i already have a patch up there | 19:31 |
| SlickNik | juice: Awesome. Thanks! | 19:32 |
| *** dkliban has quit IRC | 19:32 | |
| SergeyLukjanov | devananda, clarkb, I think that after removing neutron SMOKE_SERIAL couldbe disabled too | 19:32 |
| SlickNik | juice: link? | 19:32 |
| *** dkliban has joined #openstack-infra | 19:32 | |
| juice | https://review.openstack.org/#/c/71039/ | 19:33 |
| anteaya | SlickNik: join us in -meeting won't you? | 19:33 |
| devananda | SergeyLukjanov: when I checked with clarkb on that, my understanding was we'd have to add TEMPEST_FULL to replace it, which also seemed unnecessary | 19:33 |
| *** AlexF has joined #openstack-infra | 19:33 | |
| SergeyLukjanov | devananda, AFAIK SMOKE_SERIAL == run smoke tests in 1 thread | 19:34 |
| *** johnthetubaguy has quit IRC | 19:34 | |
| SergeyLukjanov | devananda, anyway I don't think that it's really important | 19:35 |
| *** jcooley_ has joined #openstack-infra | 19:36 | |
| *** AlexF has quit IRC | 19:36 | |
| hub_cap | dtroyer: thx for the clarification sir | 19:37 |
| *** masayukig has joined #openstack-infra | 19:37 | |
| *** pblaho has quit IRC | 19:37 | |
| shivh | fungi: hi | 19:40 |
| fungi | shivh: hi--what's up? | 19:41 |
| *** masayukig has quit IRC | 19:42 | |
| *** sdake has quit IRC | 19:42 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 19:42 |
| shivh | i dont seem to have voting enabled for the account (shivharis) | 19:42 |
| shivh | this account is exclusively for CI | 19:43 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 19:43 |
| shivh | fungi: i dont seem to have voting enabled for the account (shivharis) | 19:43 |
| *** sdake has joined #openstack-infra | 19:43 | |
| *** sdake has quit IRC | 19:44 | |
| *** sdake has joined #openstack-infra | 19:44 | |
| *** sdake has quit IRC | 19:44 | |
| *** sdake has joined #openstack-infra | 19:44 | |
| *** sdake has quit IRC | 19:44 | |
| *** sdake has joined #openstack-infra | 19:44 | |
| *** jcooley_ has quit IRC | 19:44 | |
| annegentle | SergeyLukjanov: I can chat names over here if you wanna | 19:45 |
| fungi | shivh: contributor accounts are not allowed to place verify votes. you need to request a service account for that | 19:46 |
| SergeyLukjanov | annegentle, It'll be awesome! some details: http://lists.openstack.org/pipermail/openstack-dev/2014-January/025905.html | 19:46 |
| annegentle | SergeyLukjanov: oh okay found it | 19:46 |
| *** AlexF has joined #openstack-infra | 19:46 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new job to build the javamelody gerrit plugin https://review.openstack.org/70014 | 19:47 |
| *** AlexF has quit IRC | 19:47 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 19:49 |
| *** salv-orlando has quit IRC | 19:49 | |
| *** jcooley_ has joined #openstack-infra | 19:49 | |
| *** salv-orlando has joined #openstack-infra | 19:49 | |
| *** fbo is now known as fbo_away | 19:52 | |
| shivh | fungi: where should i request a service a/c | 19:54 |
| fungi | shivh: http://ci.openstack.org/third_party.html | 19:54 |
| shivh | fungi: thanks. | 19:54 |
| *** thuc has quit IRC | 19:59 | |
| *** thuc has joined #openstack-infra | 19:59 | |
| mikal | jeblair: he should be around today, but perhaps not for another hour | 19:59 |
| *** zehicle_at_dell has quit IRC | 20:00 | |
| *** zehicle_at_dell has joined #openstack-infra | 20:00 | |
| pleia2 | ok, back to bug day? ;) | 20:00 |
| *** markmc has joined #openstack-infra | 20:00 | |
| annegentle | what is the latest on moodle, curious | 20:00 |
| clarkb | fungi: lifeless: for nodepool, it doesn't look like we kick off a delete immediately after the delete state is set | 20:01 |
| *** bhuvan_ has quit IRC | 20:01 | |
| *** sparkycollier has joined #openstack-infra | 20:01 | |
| *** emagana has joined #openstack-infra | 20:01 | |
| clarkb | the nodeCompleteThread sets teh state to DELETE and is done | 20:01 |
| clarkb | I bet that is why deletes are taking so long | 20:01 |
| jeblair | clarkb: can " Jenkins SCP Plugin fix for Elastic Search (clarkb, zaro) " be removed from the agenda? | 20:02 |
| clarkb | jeblair: yes | 20:02 |
| clarkb | we should immediately fire off a non blocking delete of the node | 20:02 |
| jeblair | clarkb: k thx; i thought that's what we talked about, just wanted to make sure | 20:02 |
| clarkb | I am going to go grab lunch now then back to bugs | 20:02 |
| fungi | annegentle: good question--i don't know | 20:03 |
| fungi | annegentle: dguitarbite and sarob wanted to toss that into the agenda at the last moment, but we were already runnign a packed meeting | 20:03 |
| *** jcooley_ has quit IRC | 20:03 | |
| *** thuc has quit IRC | 20:03 | |
| lifeless | clarkb: thats correct; the code consolidated that to avoid race conditions and now deletes them all together | 20:04 |
| lifeless | clarkb: when you say 'takes so long' - how long is it taking ? | 20:04 |
| jeblair | fungi, clarkb: btw on the plane i wrote the change to split out the zuul merger; i want to revise it a bit more but that should be up for review this week. but it passes tests. :) | 20:05 |
| *** CaptTofu has quit IRC | 20:06 | |
| SergeyLukjanov | jeblair, fungi, clarkb, could you please take a look on enabling savanna's d-g jobs? https://review.openstack.org/#/c/68066/ | 20:07 |
| *** jcooley_ has joined #openstack-infra | 20:07 | |
| fungi | clarkb: that may have been the result of https://review.openstack.org/#/c/67985/2..3/nodepool/provider_manager.py ? | 20:08 |
| *** sparkycollier has quit IRC | 20:08 | |
| jeblair | SergeyLukjanov: i moved it to the top of my list (but i can't look at it right now) | 20:08 |
| fungi | clarkb: though now that i look at it, i don't think so after all | 20:09 |
| SergeyLukjanov | jeblair, thank you! | 20:09 |
| *** jasondotstar has joined #openstack-infra | 20:09 | |
| *** rfolco has quit IRC | 20:11 | |
| *** kraman1 has joined #openstack-infra | 20:13 | |
| *** kraman has quit IRC | 20:13 | |
| *** thedodd has joined #openstack-infra | 20:14 | |
| fungi | jeblair: i'm looking forward to the new, more scalable, more better zuul. can't wait to see the patch | 20:14 |
| *** AlexF has joined #openstack-infra | 20:15 | |
| openstackgerrit | Brant Knudson proposed a change to openstack/requirements: Update keystoneclient to 0.5.0 https://review.openstack.org/71085 | 20:16 |
| *** hashar has joined #openstack-infra | 20:17 | |
| clarkb | lifeless: see graph on zuul status page | 20:17 |
| *** odyssey4me has quit IRC | 20:18 | |
| *** emagana has quit IRC | 20:20 | |
| *** sparkycollier has joined #openstack-infra | 20:22 | |
| fungi | clarkb: zaro: did we have a bug filed for the truncated console log/scp publisher issue you fixed? | 20:23 |
| *** mestery has quit IRC | 20:23 | |
| *** UtahDave has joined #openstack-infra | 20:25 | |
| *** emagana has joined #openstack-infra | 20:27 | |
| *** e0ne has joined #openstack-infra | 20:28 | |
| *** malini is now known as malini_afk | 20:29 | |
| *** _david_ has joined #openstack-infra | 20:33 | |
| *** FallenPegasus has quit IRC | 20:33 | |
| *** thuc has joined #openstack-infra | 20:35 | |
| *** atiwari has quit IRC | 20:35 | |
| wenlock | hi, was looking for a possibility of a review today https://review.openstack.org/#/c/51425 , clarkb, hoping your day is looking better | 20:36 |
| *** rakhmerov has quit IRC | 20:37 | |
| *** gokrokve has quit IRC | 20:37 | |
| *** masayukig has joined #openstack-infra | 20:38 | |
| *** gokrokve has joined #openstack-infra | 20:38 | |
| *** markmcclain has quit IRC | 20:38 | |
| *** rakhmerov has joined #openstack-infra | 20:39 | |
| fungi | jog0: have you seen https://launchpad.net/bugs/1262827 ? | 20:40 |
| *** denis_makogon_ has joined #openstack-infra | 20:41 | |
| *** weshay has quit IRC | 20:42 | |
| lifeless | clarkb: lets see | 20:42 |
| *** masayukig has quit IRC | 20:42 | |
| *** gokrokve has quit IRC | 20:43 | |
| lifeless | clarkb: whats the delete cron job schedule in config ? | 20:44 |
| *** slong has joined #openstack-infra | 20:44 | |
| *** denis_makogon_ is now known as denis_makogon | 20:44 | |
| lifeless | clarkb: -deployed that is- | 20:44 |
| *** salv-orlando has quit IRC | 20:44 | |
| *** salv-orlando has joined #openstack-infra | 20:44 | |
| *** markmcclain has joined #openstack-infra | 20:45 | |
| *** weshay has joined #openstack-infra | 20:46 | |
| jog0 | fungi: looking now | 20:48 |
| *** sdake has quit IRC | 20:48 | |
| jog0 | fungi: that has been fixed | 20:49 |
| fungi | jog0: i'm slightly hesitant to triage elastic-recheck bugs since i haven't been doing much devel/review on it | 20:49 |
| zaro | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1260983 | 20:49 |
| fungi | jog0: excellent. please update with details if you have a moment | 20:49 |
| clarkb | lifeless: */1 | 20:49 |
| clarkb | lifeless: I made the change by hand | 20:49 |
| fungi | zaro: thanks! i found (at least) one untriaged dupe of it | 20:49 |
| *** sdake has joined #openstack-infra | 20:50 | |
| *** sdake has quit IRC | 20:50 | |
| *** sdake has joined #openstack-infra | 20:50 | |
| lifeless | clarkb: so every 60s it should be picking up all the new deletes and adding them | 20:50 |
| lifeless | clarkb: is that not happening? | 20:50 |
| jog0 | fungi: I am happy to triage any e-r bugs as they get reported | 20:51 |
| clarkb | lifeless: which log message should I be looking for? | 20:52 |
| clarkb | lifeless: 'Deleting node id: .*' ? | 20:53 |
| *** thuc has quit IRC | 20:53 | |
| fungi | jog0: i'm trying to tag them with elastic-recheck as i come across them | 20:53 |
| lifeless | clarkb: Starting periodic cleanup for provider | 20:53 |
| lifeless | clarkb: Finished periodic cleanup for provider | 20:53 |
| lifeless | clarkb: get me a good section of that - say an hours worth | 20:53 |
| clarkb | lifeless: k | 20:53 |
| *** thuc has joined #openstack-infra | 20:53 | |
| jog0 | fungi: cool, is there a way I can just subscribe to that tag? | 20:53 |
| lifeless | it may be that we're blocking in there inappropriately still, which would delay submitting the deletes to the provider | 20:54 |
| * jog0 is enjoying eavesdropping on TC today | 20:54 | |
| clarkb | lifeless: how much context do you want/need? | 20:56 |
| clarkb | I have the starts and finishes if that is all you want ot start with | 20:56 |
| *** AlexF has quit IRC | 20:57 | |
| dolphm | how long of a delay should be expected between submitting a patch to gerrit with a Closes-Bug, and seeing a comment from openstack-hudson revising the bug status? (i.e. https://bugs.launchpad.net/keystone/+bug/1245629/comments/11 ) | 20:57 |
| fungi | jog0: i believe so... looks like http://askubuntu.com/questions/85410/how-do-i-subscribe-to-a-tag-in-launchpad | 20:57 |
| clarkb | dolphm: not very long, minute or two probably | 20:57 |
| clarkb | dolphm: however there are cases where it will not spam the bug | 20:58 |
| dolphm | clarkb: how about this one https://bugs.launchpad.net/keystone/+bug/1276325 https://review.openstack.org/#/c/71096/ | 20:58 |
| fungi | dolphm: clarkb: it's broken https://launchpad.net/bugs/1276180 | 20:58 |
| *** thuc has quit IRC | 20:58 | |
| fungi | filed this morning, haven't had time to dig deeper yet | 20:58 |
| dolphm | oh there we go | 20:58 |
| *** atiwari has joined #openstack-infra | 20:58 | |
| lifeless | clarkb: lets start with that | 20:58 |
| fungi | trimmed those out of the gerrit error.log | 20:58 |
| *** nati_ueno has quit IRC | 20:59 | |
| jog0 | fungi: neato that was easy | 20:59 |
| *** sparkycollier has quit IRC | 21:00 | |
| fungi | jog0: i made elastic-recheck an official bug tag for openstack-ci so that should help wrangle those | 21:00 |
| *** sparkycollier has joined #openstack-infra | 21:00 | |
| *** nati_ueno has joined #openstack-infra | 21:00 | |
| jog0 | fungi: any reason to use fix commited vs fix released | 21:01 |
| fungi | jog0: if the project the corresponding patch went into does releases, then fix committed when the fix is committed, and fix released once the next release containing that commit is cut | 21:02 |
| *** sdake has quit IRC | 21:02 | |
| fungi | if there are never releases, then just straight to fix released | 21:02 |
| jog0 | ack | 21:03 |
| clarkb | fungi: we seem to have removed the --- doc separator from our yaml projects file | 21:03 |
| *** dizquierdo has joined #openstack-infra | 21:03 | |
| fungi | clarkb: right, i suspect https://launchpad.net/bugs/1275824 is also in the same boat | 21:04 |
| clarkb | fungi: ya | 21:05 |
| jeblair | fungi: what does 'elastic-recheck' tag mean? | 21:05 |
| jeblair | fungi: and how does it compare to 'gate-failure'? | 21:06 |
| fungi | bits of jeepyb need to do a better job of autodetecting whether the config and project registry are split out or not. i originally had thought this would get done with config format versioning, but instead i guess we're just guessing based on file existence | 21:06 |
| SergeyLukjanov | clarkb, you've mentioned that -check slaves could be used for -nv, what's the difference? performance? | 21:06 |
| jeblair | fungi: oh, bugs on the elastic-recheck program? | 21:06 |
| jeblair | heh, i mean 'project'. :) | 21:06 |
| clarkb | SergeyLukjanov: yes, we run extra clouds for those slaves | 21:07 |
| clarkb | SergeyLukjanov: and is the reason check vs gate is a thing in the job names | 21:07 |
| *** gokrokve has joined #openstack-infra | 21:08 | |
| clarkb | fungi: ya, I will propose the --- back into the file then we can work on fixing jeepyb. Just testing that yaml does the right thing if [0] is nil | 21:08 |
| fungi | jeblair: elastic-recheck is for bugs in elastic recheck itself | 21:08 |
| fungi | "elastic recheck did this thing i did not expect" | 21:08 |
| fungi | et cetera | 21:08 |
| * fungi nods | 21:08 | |
| *** resker has quit IRC | 21:08 | |
| *** zehicle_at_dell has quit IRC | 21:08 | |
| fungi | if we get too many bugs about elastic-recheck reported against openstack-ci, we could always just add a new launchpad project (or storyboard group!) for them | 21:08 |
| *** esker has joined #openstack-infra | 21:08 | |
| *** mrda_away is now known as mrda | 21:09 | |
| fungi | clarkb: at least manage-projects will go toes up in that case. it looks to see if the ini file exists and then expects the yaml projects list to be in the first (only) document of the yaml file | 21:09 |
| *** sparkycollier has quit IRC | 21:09 | |
| SergeyLukjanov | clarkb, ok, got it, it should be used only for check/non-voting jobs? | 21:09 |
| clarkb | fungi: gah nevermind then, I will propose that fix to jeepyb instead | 21:09 |
| jeblair | fungi: ack. yep. | 21:09 |
| clarkb | SergeyLukjanov: yes | 21:09 |
| *** gokrokve_ has joined #openstack-infra | 21:10 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests https://review.openstack.org/68066 | 21:11 |
| fungi | clarkb: i've updated the bug with a quick redux of what manage-projects is doing to make that determination | 21:12 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Allow devstack-precise-check for non-voting jobs https://review.openstack.org/71103 | 21:13 |
| SergeyLukjanov | clarkb, the same for ironic ^^ | 21:13 |
| *** gokrokve has quit IRC | 21:13 | |
| *** oubiwann_ has quit IRC | 21:13 | |
| *** gokrokve_ has quit IRC | 21:14 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove tripleo check jobs from devstack-gate https://review.openstack.org/71104 | 21:15 |
| jeblair | clarkb, fungi: can you review/aprv ^ soon please? | 21:16 |
| fungi | jeblair: already +2'd | 21:16 |
| *** mattoliverau has joined #openstack-infra | 21:17 | |
| *** sdake has joined #openstack-infra | 21:17 | |
| *** sdake has quit IRC | 21:17 | |
| *** sdake has joined #openstack-infra | 21:17 | |
| *** dkliban has quit IRC | 21:17 | |
| fungi | i discussed the possibility with lifeless yesterday as well | 21:17 |
| anteaya | morning mattoliverau, jeblair was looking for you | 21:17 |
| mattoliverau | Good morning anteaya. was he | 21:17 |
| clarkb | approved | 21:17 |
| mattoliverau | morning jeblair | 21:18 |
| *** sdake has quit IRC | 21:19 | |
| *** lcestari has quit IRC | 21:19 | |
| *** sdake has joined #openstack-infra | 21:19 | |
| *** sdake has quit IRC | 21:19 | |
| *** sdake has joined #openstack-infra | 21:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove tripleo check jobs from devstack-gate https://review.openstack.org/71104 | 21:19 |
| *** emagana has quit IRC | 21:20 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests https://review.openstack.org/68066 | 21:20 |
| jeblair | fungi: yeah, it's problematic now, but would be even more so if we require check jobs for gate enqueing | 21:20 |
| jeblair | mattoliverau: hey! i have an idea for something you could work on if you have some time; fairly self-contained, mostly puppet, touches a bit more of the infra than your last change | 21:21 |
| fungi | full agreement | 21:21 |
| *** sarob has quit IRC | 21:22 | |
| mattoliverau | jeblair: great! what would you like? | 21:22 |
| *** alexpilotti has quit IRC | 21:22 | |
| *** CaptTofu has joined #openstack-infra | 21:22 | |
| jeblair | mattoliverau: there's this program called nodepool and it's running on its own server (called nodepool.o.o), which is pretty much a black box. one of the things it does is build vm images for the machines that are used to run tests | 21:22 |
| jeblair | mattoliverau: it logs those image builds to files, but they are inaccessible without logging into the server | 21:23 |
| jeblair | mattoliverau: it would be cool if they were logged to a place that could be served by apache | 21:23 |
| jeblair | mattoliverau: so anyone could see the logs | 21:23 |
| jeblair | fungi, clarkb: i'm pretty sure nothing in the image build logs is security sensitive... right? | 21:23 |
| fungi | i take it we're comfortable that exceptions in nodepool won't leak provider credentials? | 21:24 |
| *** freyes has quit IRC | 21:24 | |
| fungi | we also get a full dump of the puppet apply stdout/stderr in the image log... could that include diffs of sensitive files? | 21:24 |
| jeblair | fungi: i believe that the image logs only log the output from running the scripts; and it could contain sensitive info if there were any, but an easier way to get sensitive info off a devstack host would be to write a change to 'cat' it and submit to gerrit | 21:25 |
| *** thuc has joined #openstack-infra | 21:25 | |
| jeblair | fungi: so for that purpose, we don't put any sensitive info on those hosts | 21:25 |
| fungi | jeblair: very, very true | 21:26 |
| jeblair | s/purpose/reason/ | 21:26 |
| * fungi is trying to think up any other devil's advocate concerns | 21:26 | |
| *** Ajaeger has quit IRC | 21:26 | |
| jeblair | ya | 21:26 |
| fungi | i'm skimming the image log and nothing is jumping out at me screaming "not on the web!" | 21:27 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Adding query for bug 1275500 https://review.openstack.org/70756 | 21:27 |
| mattoliverau | jeblair: Ok, consider it done :) I'll take a look, but if you do find something sensitive just let me know, or deny the patch when it comes in ;) | 21:27 |
| fungi | though ick... tons of embedded carriage returns without newlines | 21:27 |
| *** miqui has quit IRC | 21:28 | |
| fungi | seems to be progress output from some tools which think they're in an interactive shell | 21:28 |
| fungi | wonder if that's a new side effect of the pty patch | 21:28 |
| *** emagana has joined #openstack-infra | 21:28 | |
| jeblair | mattoliverau: a follow-up patch would be to improve the logging for nodepool; that's mostly python hacking. let me know if you're interested. | 21:29 |
| clarkb | fungi: so this is extra fun because projects.py seems to change the type of the projects list... | 21:29 |
| mattoliverau | jeblair: sure am :) | 21:29 |
| mattoliverau | jeblair: thanks for thinking of me | 21:29 |
| clarkb | fungi: they are in an interactive shell :) | 21:30 |
| jeblair | mattoliverau: btw, i think this is going to be really useful -- we want the infra to be as accessible to non-root people as possible, and this will help that a lot, including tripleo folks who are working on image bootstrapping right now | 21:30 |
| clarkb | because sudo | 21:30 |
| *** amcrn has quit IRC | 21:30 | |
| *** sarob has joined #openstack-infra | 21:30 | |
| *** changbl has quit IRC | 21:31 | |
| *** jcooley_ has quit IRC | 21:31 | |
| *** DinaBelova is now known as DinaBelova_ | 21:32 | |
| mattoliverau | great, well I'd better get on it then :) | 21:34 |
| *** emagana has quit IRC | 21:34 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs. https://review.openstack.org/71110 | 21:36 |
| clarkb | fungi: ^ | 21:36 |
| clarkb | ok /me goes back to poking bugs | 21:37 |
| markmcclain | the gate reset… would it possible to promote https://review.openstack.org/#/c/71046/ to the top incase it happens again | 21:38 |
| *** masayukig has joined #openstack-infra | 21:39 | |
| *** mestery has joined #openstack-infra | 21:39 | |
| clarkb | markmcclain: sure, did we just reset now? | 21:39 |
| *** mestery has quit IRC | 21:40 | |
| markmcclain | yes | 21:40 |
| fungi | markmcclain: clarkb looks like the nova job is reporting its fail right this moment | 21:40 |
| fungi | and there it went | 21:40 |
| fungi | so we should be safe to promote | 21:40 |
| *** mestery has joined #openstack-infra | 21:40 | |
| fungi | clarkb: i've got the command line ready in an ssh session on zuul | 21:41 |
| clarkb | fungi: go for it | 21:41 |
| fungi | just want to avoid both doing it | 21:41 |
| fungi | done | 21:41 |
| fungi | it's at the top now | 21:41 |
| fungi | markmcclain: ^ | 21:41 |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1172384 is that something we can close? | 21:42 |
| mgagne | In gerrit, must all change topics be unique across the branch? Can't I provide the same topic twice if the first one is closed/merged? | 21:42 |
| markmcclain | fungi: thank you | 21:42 |
| clarkb | mgagne: you should be able to reuse topics | 21:42 |
| *** masayukig has quit IRC | 21:43 | |
| jeblair | clarkb: sure, there's more work there but i think we can close it | 21:43 |
| jeblair | clarkb: (i closed it) | 21:43 |
| clarkb | jeblair: ya definitely more work, but it is flagging things today | 21:43 |
| annegentle | oh yeah, so moodle. where / what domain/ subdomain? | 21:44 |
| mgagne | clarkb: thanks, I'll check what's going on with my issue | 21:44 |
| annegentle | oh, sorry fungi already answered that we don't really know yet :) | 21:44 |
| clarkb | http://logs.openstack.org/10/71110/1/check/gate-jeepyb-pep8/2536094/console.html that has me stumped. | 21:44 |
| *** dkliban_afk has joined #openstack-infra | 21:44 | |
| jeblair | annegentle: can you catch me up on your question? i've been away | 21:45 |
| clarkb | moodle? | 21:45 |
| mgagne | clarkb: I reverted a backport and I'm trying to cherry-pick it again. Gerrit won't let me as it would submit the same Change-Id twice | 21:45 |
| *** dprince has quit IRC | 21:45 | |
| *** sarob has quit IRC | 21:45 | |
| clarkb | mgagne: change ids must be different | 21:45 |
| clarkb | topics don't need to be | 21:45 |
| mgagne | clarkb: ok, so I change to generate a new one right? | 21:46 |
| clarkb | mgagne: yup | 21:46 |
| mgagne | clarkb: ok, thanks | 21:46 |
| fungi | jeblair: dguitarbite added a last-minute agenda item about moodle app integration for the training manuals \to the meeting today but wasn't around at the end and we wouldn't have had time to discuss it | 21:47 |
| fungi | apparently sarob was also mentioned on it but currently isn't around either | 21:48 |
| fungi | no idea what "Request for Moodle App Integration to infra for Training-Manuals" means (and i'm doing my best not to ponder the possibilities) | 21:49 |
| clarkb | I see the problem with jeepyb pep8 checks. pbr in new hacking conflicts with pbr elsewhere. I am updating requirements for jeepyb | 21:50 |
| *** gyee has quit IRC | 21:50 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1275256 https://review.openstack.org/71118 | 21:51 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Fix pbr and hacking requirements. https://review.openstack.org/71119 | 21:51 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs. https://review.openstack.org/71110 | 21:51 |
| clarkb | I think that should get us sorted | 21:51 |
| annegentle | jeblair: fungi: clarkb: yeah adding moodle seems like a lot of work but want to hear what they need it for. probably quizzes? class graduation tracking? Not sure what other requirements they have? | 21:52 |
| clarkb | annegentle: but why is that a thing we would be involved in? | 21:53 |
| annegentle | clarkb: because they'd need it stood up, domain registered, etc. I guess | 21:53 |
| fungi | clarkb: maybe publishing stuff to moodle | 21:53 |
| clarkb | wait is the foundation certifying people direclty? | 21:53 |
| annegentle | they're not really going through me for this | 21:53 |
| annegentle | as far as requirement gathering | 21:53 |
| fungi | clarkb: i don't think so, no | 21:53 |
| annegentle | clarkb: no idea | 21:53 |
| *** jhesketh has joined #openstack-infra | 21:54 | |
| *** apevec has quit IRC | 21:54 | |
| fungi | clarkb: i believe the foundation is helping the community settle on standards for certification tests | 21:54 |
| jhesketh | Morning | 21:54 |
| anteaya | morning jhesketh | 21:54 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1265495 https://review.openstack.org/70556 | 21:55 |
| mriedem | jog0: ^ | 21:55 |
| *** sdake has quit IRC | 21:55 | |
| fungi | clarkb: but as far as i know we're not going to be endorsing any one particular test provider/company and want the materials to be free as in speech | 21:55 |
| clarkb | gotcha | 21:56 |
| jog0 | mriedem: mriedem I am having trouble keeping up with you now | 21:56 |
| clarkb | woot that fixed jeepyb tests | 21:56 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1275256 https://review.openstack.org/71118 | 21:56 |
| mriedem | jog0: that's ok, i'm out of here in about 35 minutes anyway :) | 21:56 |
| clarkb | now I can actually move on to more bug stuff | 21:56 |
| *** nicedice has quit IRC | 21:56 | |
| *** nicedice has joined #openstack-infra | 21:57 | |
| jog0 | mriedem: heh, once your patches land hopefully our classifcation rate will jump http://status.openstack.org/elastic-recheck/data/uncategorized.html | 21:59 |
| *** hashar has quit IRC | 21:59 | |
| mriedem | oh yeah, that | 21:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1265495 https://review.openstack.org/70556 | 22:01 |
| *** masayukig has joined #openstack-infra | 22:01 | |
| *** markmc has quit IRC | 22:01 | |
| *** UtahDave has quit IRC | 22:03 | |
| *** e0ne has quit IRC | 22:04 | |
| *** david-lyle has quit IRC | 22:04 | |
| clarkb | I am closing https://bugs.launchpad.net/openstack-ci/+bug/1172443 beause infra projects use testr and there is a bug with the other projects in it for tracking progress across most projects | 22:04 |
| fungi | k | 22:04 |
| *** jasondotstar has quit IRC | 22:05 | |
| *** masayukig has quit IRC | 22:05 | |
| *** whoops has joined #openstack-infra | 22:08 | |
| clarkb | fungi: https://review.openstack.org/#/c/53245/1 :( | 22:09 |
| clarkb | we can close that bug but it got auto abandoned during summit and missed | 22:10 |
| clarkb | jeblair: ^ if you guys want to review that really quick it would be awesome. I am checking it for merge conflicts now | 22:10 |
| *** jcooley_ has joined #openstack-infra | 22:10 | |
| clarkb | it conflicts, fixing now | 22:12 |
| mordred | clarkb: I just -1'd 71159 | 22:12 |
| mordred | clarkb: I just -1'd 71119 - there is no 71159 | 22:12 |
| mordred | clarkb: I'm happy to just fix it instead though | 22:12 |
| clarkb | mordred: go for it | 22:13 |
| fungi | yeah, it'll need a rebase | 22:13 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Install graphiz on Jenkins slaves. https://review.openstack.org/53245 | 22:13 |
| mordred | clarkb: k. /me wants to at least attempt to be useful | 22:13 |
| jeblair | clarkb: i -1 53245 due to a typo | 22:13 |
| jeblair | clarkb: (which is also in the commit msg, btw) | 22:13 |
| jeblair | clarkb: but i did not -1 it because of the commit msg | 22:14 |
| fungi | i had totally forgotten pbr needed to be first in the reqs lists as well | 22:14 |
| clarkb | jeblair: good catch | 22:14 |
| clarkb | jeblair: actually something else may have fixed that in the meantime | 22:15 |
| clarkb | auto abandoner makes me not happy right now | 22:15 |
| clarkb | jeblair: I will abandon 53245 and close the bug. A different change added graphviz to the slaves | 22:15 |
| *** sdake has joined #openstack-infra | 22:16 | |
| fungi | i'm completely in favor of turning it off until we have new gerrit, and then looking into auto-wip | 22:16 |
| *** jcooley_ has quit IRC | 22:16 | |
| *** _david_ has quit IRC | 22:16 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs. https://review.openstack.org/71110 | 22:17 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Fix pbr and hacking requirements. https://review.openstack.org/71119 | 22:17 |
| mordred | clarkb: I'm good with both of those now | 22:17 |
| *** jcooley_ has joined #openstack-infra | 22:17 | |
| clarkb | mordred: thanks | 22:17 |
| anteaya | mordred: I'm helping you with the manage projects script | 22:18 |
| anteaya | when are you available to work on it? | 22:18 |
| * anteaya gets out her butterfly net | 22:19 | |
| mordred | anteaya: heh.consider me unavailable this week | 22:20 |
| anteaya | so that means next Monday | 22:20 |
| anteaya | did I hear that correctly? | 22:20 |
| *** CaptTofu has quit IRC | 22:23 | |
| clarkb | mordred: https://bugs.launchpad.net/openstack-ci/+bug/1179007 how are we doing there? | 22:23 |
| *** mestery has quit IRC | 22:24 | |
| mordred | clarkb: I'd need to go explicitly verify things | 22:25 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Fix pbr and hacking requirements. https://review.openstack.org/71119 | 22:25 |
| *** amcrn has joined #openstack-infra | 22:25 | |
| *** markmcclain has quit IRC | 22:26 | |
| clarkb | mordred: and https://bugs.launchpad.net/openstack-ci/+bug/1179008 goes along with that. | 22:27 |
| clarkb | how do people think about marking https://bugs.launchpad.net/openstack-ci/+bug/1182231 won't fix? I believe our clouds will have heat for us in the enearish future which is far better tha nrunning our own | 22:28 |
| clarkb | reed: https://bugs.launchpad.net/openstack-ci/+bug/1182690 is there an update for that? I don't know paul's nick | 22:28 |
| jeblair | zaro: can you look at https://bugs.launchpad.net/openstack-ci/+bug/1276293 ? | 22:28 |
| jeblair | zaro: i think i saw something related to that in review recently, but i don't recall the details | 22:29 |
| clarkb | reed: iirc the italian stuff happened? can we close that bug? | 22:29 |
| zaro | jeblair: sure. assigned to myself | 22:30 |
| jeblair | zaro: to be clear, i wasn't necessarily asking you to fix it, just thought you might know something about it | 22:30 |
| zaro | jeblair: ohh. then no. i haven't noticed that it was failing. i'll double check off master. | 22:31 |
| lifeless | clarkb: so it looks to me like it is aggressively cleaning up | 22:32 |
| lifeless | clarkb: can you also pull out 'Deleting node id:' and 'Exception deleting node id:' lines for me | 22:33 |
| jeblair | zaro: https://review.openstack.org/#/c/63579/ is the change i was thinking of | 22:33 |
| clarkb | lifeless: yup | 22:33 |
| zaro | jeblair: epy27 and epy26 both worked fine. | 22:33 |
| lifeless | clarkb: with the starting/finished periodic lines | 22:33 |
| *** markmcclain has joined #openstack-infra | 22:33 | |
| fungi | since that's the last untriaged bug, i'm going to take a quick dinner break before my next meeting | 22:34 |
| jeblair | fungi, clarkb, lifeless: i wonder if the full nodepool logs are safe to serve publicly, or is there a chance they could have creds? | 22:34 |
| lifeless | clarkb: from my quick scan there is at most 120s latency from delete to delete attempt [worst case] and I'd expect average of 45-50s | 22:34 |
| zaro | jeblair: hmm, wondering if it's because the jjb cache. maybe works on jenkins because jenkins build always has a fresh ENV. | 22:36 |
| zaro | i'll do a little test. | 22:36 |
| *** dims has quit IRC | 22:39 | |
| clarkb | mordred: https://bugs.launchpad.net/openstack-ci/+bug/1224705 | 22:40 |
| *** dkranz has quit IRC | 22:41 | |
| lifeless | jeblair: clarkb: so we're deleting nodes after ~12 seconds | 22:41 |
| mordred | clarkb: we need to get wheels uploading to pypi before we can fix that | 22:41 |
| clarkb | mordred: speaking of wheels I could not reproduce the wheel failures we saw when we enabled wheels | 22:42 |
| *** mriedem has quit IRC | 22:42 | |
| clarkb | also I hate being mean but I think we should unpin tox | 22:42 |
| lifeless | jeblair: clarkb: whats worrying though is that some of them are much longer | 22:42 |
| clarkb | there is no traction to fix that right now | 22:43 |
| lifeless | like .2 of an hour | 22:43 |
| clarkb | lifeless: yeah clearly there are nodes taking longer, possibly due to the provider side of things | 22:43 |
| mordred | clarkb: that's excellent | 22:43 |
| mordred | clarkb: what breaks when we do? | 22:43 |
| clarkb | mordred: posargs apparently? but that shouldn't break and pip does not allow external with new tox | 22:43 |
| jeblair | lifeless: yes, some providers take a long time; that's less worrying since it's previously observed behavior; it's definitely worring from the perspective of "openstack should be good" | 22:44 |
| mordred | clarkb: I'm in favor of unpinning - I looked at the new docs and couldn't figure out how our usages would be broken | 22:44 |
| clarkb | mordred: the virtualenv and pip upgrades are the biggest headache | 22:44 |
| lifeless | clarkb: jeblair: no, you misunderstand me | 22:44 |
| clarkb | mordred: it will break and the gate will grind to a halt I guaruntee it | 22:45 |
| lifeless | clarkb: jeblair: in that log file, the *first* occurence of the node id in the delete routine is when it's been in that state for many minutes. | 22:45 |
| mordred | clarkb: then we shouuld maybe not do it yet | 22:45 |
| clarkb | mordred: I would agree with you if we had fixed sqlalchemy by now | 22:45 |
| lifeless | Uhm, I think I need to see what provider they are on too; prepping a patch for that | 22:46 |
| *** mfer has quit IRC | 22:46 | |
| jeblair | lifeless: oh yes i misunderstood, sorry. | 22:46 |
| anteaya | mattoliverau: oh, you probably haven't participated in a bug day yet, today is bug day | 22:47 |
| *** eharney has quit IRC | 22:47 | |
| anteaya | mattoliverau: during bug day someone (pleia2) creates an etherpad for the event and populates it with our bugs for ci: https://etherpad.openstack.org/p/cibugreview-february2014 | 22:47 |
| dhellmann | anteaya: is today a good day to ask about importing some git repos, or should I wait until tomorrow? | 22:47 |
| *** jhesketh__ has joined #openstack-infra | 22:47 | |
| anteaya | dhellmann: asking is always good | 22:47 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Also log provider name when debugging deletes. https://review.openstack.org/71137 | 22:48 |
| anteaya | expecting them to be auto managed won't happen today, or apparently the rest of this week | 22:48 |
| anteaya | but right now mordred is hand creating them | 22:48 |
| lifeless | jeblair: if I can get ^ [untested] on prod and some data, we can see if its slow cleanup loops (in which case optimising that will be the next step) or if its something unexpected. | 22:48 |
| anteaya | so perhaps if a patch is merged, he can do that for you | 22:48 |
| clarkb | fungi: was there any additional breakage from new tox? I wasn't awake for most of it | 22:48 |
| dhellmann | anteaya: hmm, does mordred still have a fever? ;-) | 22:48 |
| anteaya | mordred: do you still have a fever? | 22:49 |
| dhellmann | if you guys are working on tools, these repos can wait | 22:49 |
| anteaya | well you guys right now is me | 22:49 |
| lifeless | adding in an instant-try-delete-now will a) add race conditions with sqla you wanted to avoid and b) increase API pressure which will slow everything down by ratelimiting use in a more random fashion | 22:49 |
| *** gokrokve has joined #openstack-infra | 22:49 | |
| clarkb | fungi: I thought https://bugs.launchpad.net/openstack-ci/+bug/1263160 required non open software | 22:49 |
| anteaya | and I am slow at the best of times, and trying to come down off of neutron and refocus on -infra again | 22:49 |
| dhellmann | anteaya: I have a patch up and dims has another one, so they won't be forgotten | 22:49 |
| anteaya | dhellmann: what is your timeline for the repos? | 22:49 |
| anteaya | good | 22:49 |
| lifeless | the whole point of grouping this up is to ensure we scan across all the deletes at once so that we don't block other threads (by the ratelimiting aspect) and thus starve deletes that could happen quickly, perhaps. | 22:50 |
| anteaya | if mordred gets them in by hand, that is good, if I beat him to it, even better | 22:50 |
| dhellmann | anteaya: soon is nice, no particular rush, yet | 22:50 |
| dhellmann | anteaya: I'll add you to the reviews | 22:50 |
| anteaya | dhellmann: awesome thanaks | 22:51 |
| anteaya | thanks | 22:51 |
| zaro | jeblair: i could not repro, i updated the bug to get more info. have assigned it to me so i'll keep an eye out. | 22:51 |
| clarkb | mordred: do you remember how we enabled wheels back when we neabled it? where did the pip flag to use wheels get enabled? | 22:52 |
| clarkb | mordred: I can only find the change to select-mirror to list distro specific mirror locations | 22:52 |
| clarkb | mordred: was that enabled in pbr? | 22:53 |
| *** dims_ has joined #openstack-infra | 22:55 | |
| *** pcrews has quit IRC | 22:55 | |
| *** weshay has quit IRC | 22:55 | |
| *** thuc has quit IRC | 22:55 | |
| anteaya | mattoliverau: so we go through them, id the ones we are working on, update the etherpad and the bug report so the status is current, and those that are able to, triage bugs | 22:55 |
| mattoliverau | anteaya: no I have not. Cool so there is a big list of bugs. | 22:56 |
| anteaya | mattoliverau: and we discuss the ones that need to be discussed and track down anyone we need to hear from | 22:56 |
| *** thuc has joined #openstack-infra | 22:56 | |
| mattoliverau | ahh i see | 22:56 |
| anteaya | mattoliverau: we have a bug day once per milestone | 22:56 |
| anteaya | usually just after the milestone cut | 22:56 |
| anteaya | anyone can organize, pleia2 has been driving this ship for a bit, and doing a great job at it | 22:57 |
| *** mfer has joined #openstack-infra | 22:57 | |
| *** mfer has quit IRC | 22:57 | |
| *** pcrews has joined #openstack-infra | 22:58 | |
| reed | clarkb, the bug is still valid | 22:59 |
| reed | clarkb, paul's nick is rockstar | 22:59 |
| reed | clarkb, going to bed now, I'm in Europe | 22:59 |
| *** thuc has quit IRC | 23:00 | |
| *** dizquierdo has quit IRC | 23:01 | |
| anteaya | mattoliverau: so next time we have a bug day chat with pleia2 about the date, she posts to the infra ml: http://lists.openstack.org/pipermail/openstack-infra/2014-January/000734.html | 23:01 |
| *** harlowja is now known as harlowja_away | 23:01 | |
| anteaya | reed sleep well | 23:02 |
| mattoliverau | anteaya: ok, thanks for the info. got a team meeting, bbs. | 23:03 |
| anteaya | enjoy | 23:03 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Create git repo and jenkins jobs for oslo.vmware https://review.openstack.org/70761 | 23:03 |
| mattoliverau | thanks | 23:03 |
| *** burt1 has quit IRC | 23:04 | |
| anteaya | I'm about to make an aweful suggestion | 23:04 |
| clarkb | mordred: ya I am configuring my local pip.conf to use the wheel locations and I cannot get tox 1.6.1 to fail like it did back then | 23:05 |
| anteaya | what would happen if we moved the infra meeting to 2200 utc on Tuesdays? | 23:05 |
| anteaya | it would no longer conflict wtih tripleo | 23:05 |
| clarkb | mordred: also because we use deps in tox we won't actuall use wheels until pip 1.5 | 23:05 |
| anteaya | and it hopefully would allow mattoliverau to attend | 23:05 |
| anteaya | if he is infra, and he is, he should at least have a fighting chance to attend team meetings | 23:06 |
| clarkb | anteaya: does it conflict with anything else? | 23:06 |
| *** krotscheck_sick has quit IRC | 23:06 | |
| anteaya | well the -meeting channel is empty after project meeting | 23:07 |
| anteaya | let me check teh schedule | 23:07 |
| clarkb | anteaya: some meetings happen every other week | 23:07 |
| clarkb | I thought tuesday afternoons are completely booked | 23:07 |
| fungi | clarkb: on 1263160, possibly so... i was just breaking out individual bugs for the remaining non-nova servers | 23:07 |
| clarkb | gotcha | 23:07 |
| anteaya | clarkb: ah, let me look with an eye for alternate weeks | 23:08 |
| clarkb | fungi: mordred: I am trying really hard to make wheels fail for me locally running tox. No luck so far | 23:08 |
| *** branen_ has quit IRC | 23:08 | |
| anteaya | I just control-f'd through the meetings wikipage filtering on Tuesday | 23:10 |
| clarkb | I am guessing mordred went to sleep | 23:10 |
| anteaya | noone has 2200 utc that I saw | 23:11 |
| clarkb | anteaya: cool | 23:11 |
| *** jgrimm_ has quit IRC | 23:11 | |
| *** masayukig has joined #openstack-infra | 23:11 | |
| *** prad_ has quit IRC | 23:12 | |
| anteaya | I'll put something on next week's agenda about changing the meeting time | 23:12 |
| clarkb | fungi: are the only issues with new tox that virtualenv and pip are upgraded as well? | 23:13 |
| anteaya | I wonder if I should but a placeholder somewhere that we are considering this time slot | 23:13 |
| clarkb | fungi: I can't find anything in new tox itself that should actually be a problem | 23:13 |
| anteaya | so it doesn't get filled while we are discussing it | 23:13 |
| *** harlowja_away is now known as harlowja | 23:13 | |
| clarkb | the posargs thing is weird I guess I debug that first | 23:14 |
| *** dstanek has quit IRC | 23:14 | |
| *** jecarey has quit IRC | 23:14 | |
| clarkb | oh https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror that is amazing | 23:15 |
| clarkb | now why does testr-args require the = | 23:15 |
| *** masayukig has quit IRC | 23:15 | |
| mordred | clarkb: I think the other problem with wheels may have had to do with the thing that was why pip 1.5 broke | 23:15 |
| *** dcramer_ has quit IRC | 23:16 | |
| clarkb | mordred: pip 1.5 broke beacuse it disabled externally hosted reqs | 23:16 |
| clarkb | 1.4 didn't | 23:16 |
| mordred | there was another thing too | 23:16 |
| mordred | which had to do witha bug in wheels and bootstrapping into setuptools wheels | 23:16 |
| *** mestery has joined #openstack-infra | 23:17 | |
| clarkb | mordred: where is the python setup.py test entrypoint installed? is that pbr? | 23:17 |
| *** mestery has quit IRC | 23:17 | |
| mordred | I _believe_ that the reason I was baffled by the wheels work before was because it was hitting that bug, but I thought it was a bug in our wheels work | 23:17 |
| mordred | while I _think_ it was an upstream bug | 23:17 |
| mordred | clarkb: yes. pbr | 23:17 |
| *** mestery has joined #openstack-infra | 23:17 | |
| *** sarob has joined #openstack-infra | 23:18 | |
| clarkb | mordred: why is the option testr-args= there? | 23:18 |
| mordred | it's not an entrypoint though really - when pbr is running setup.py, it injects commands more directly into setuptools | 23:18 |
| mordred | because ... uhm ... | 23:18 |
| clarkb | the = is what breaks tox 1.7 | 23:18 |
| *** zehicle_at_dell has joined #openstack-infra | 23:18 | |
| mordred | wow. really? | 23:18 |
| clarkb | mordred: yes https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror | 23:18 |
| clarkb | well I mean if we could put whitespace after the = it would be fine but that is just awkward | 23:18 |
| clarkb | instead I think we want --testr-args '{posargs}' | 23:19 |
| fungi | clarkb: catching up... the issue we ran into with new tox is that it requires newer virtualenv/pip than we were ready for | 23:19 |
| anteaya | I have added it to the agenda and edited in a placeholder as well, which I will free up if we decide against the proposal | 23:19 |
| mordred | I think it's there so taht we could do something fancy ... | 23:20 |
| clarkb | mordred: fungi: I think untangling both wheels and tox is very important and in the future I think we should consider leaving things unpinned for longer | 23:20 |
| *** thuc_ has joined #openstack-infra | 23:21 | |
| clarkb | mordred: does -t work? is that what the 't' there means? | 23:21 |
| fungi | clarkb: i'm cool with tox unpinning... that last one cropped up while only i was around so i didn't have time to dig into that and everything else that was going on | 23:21 |
| clarkb | fungi: right I think we leave it unpinned so that others dig into it :) | 23:21 |
| anteaya | mattoliverau: would Tuesdays at 2200 utc allow you to participate in an -infra team meeting? | 23:22 |
| mordred | clarkb: oh - maybe? | 23:22 |
| fungi | we'll need to unpin tox/virtualenv/pip together though | 23:22 |
| anteaya | I guess I should check with him | 23:22 |
| mordred | clarkb: yes. -t shoudl work | 23:22 |
| clarkb | mordred: so I think we remove the = from pbr and/or switch to using -t | 23:23 |
| clarkb | but none of that matters until pip and virtualenv work | 23:23 |
| mattoliverau | anteaya: yeah, that works well for me! | 23:23 |
| *** thomasem has quit IRC | 23:24 | |
| mattoliverau | thanks | 23:24 |
| anteaya | well so far it is proposed | 23:24 |
| mordred | clarkb: the = is part of setuptools parsing - we can't fix it | 23:24 |
| mordred | clarkb: so - I think that we should move to -t | 23:24 |
| mordred | and before that, we should do wheel uploads | 23:24 |
| anteaya | mattoliverau: let's see what others have to say when it is discussed | 23:24 |
| mordred | then fix the proejct that's still using direct urls | 23:24 |
| mordred | then do pbr/pip/virtualenv | 23:25 |
| anteaya | but you will learn more the more you can participate | 23:25 |
| mordred | we can do the -t fix everywhere and it shoudl work old and new, btw | 23:25 |
| mattoliverau | anteaya: yeah fair enough :) | 23:25 |
| *** markmcclain has quit IRC | 23:25 | |
| anteaya | mattoliverau: :D | 23:25 |
| pleia2 | zaro: want to flip the switch on bug 1276293 to "Incomplete" (that's the last of our "New" bugs!) | 23:25 |
| *** rcleere has quit IRC | 23:26 | |
| fungi | mordred: i think we needed oslo.config to have an initial release to pypi too | 23:26 |
| fungi | no, wasn't oslo.config | 23:27 |
| *** sarob_ has joined #openstack-infra | 23:27 | |
| *** rfolco has joined #openstack-infra | 23:27 | |
| mordred | oslo.messaging | 23:27 |
| clarkb | mordred: right so, I suggest unpinning beacuse otherwise none of that stuff will happen before icehouse | 23:27 |
| fungi | oslo.messaging, but it seems to have released finally | 23:27 |
| clarkb | :) | 23:27 |
| mordred | oh - wait | 23:27 |
| fungi | that | 23:27 |
| clarkb | it is mean, but it is reality | 23:27 |
| mordred | they've already released | 23:27 |
| mordred | which means can change the reqs | 23:28 |
| fungi | it's an alpha release, but at least it's uploaded | 23:28 |
| mordred | that's all we need | 23:28 |
| *** miqui has joined #openstack-infra | 23:28 | |
| fungi | so yes, we need reqs changed and then projects can fix themselves to match | 23:28 |
| *** sarob has quit IRC | 23:28 | |
| *** mriedem has joined #openstack-infra | 23:29 | |
| clarkb | weren't there also some straggler reqs externally hosted that were not oslo.*? | 23:29 |
| *** mriedem has quit IRC | 23:29 | |
| clarkb | the problem with this is these are all project things, and we end up fixing them because we pinned | 23:30 |
| fungi | there are, but the projects using them hopefully have their tox.ini files patched now | 23:30 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Map failed jobs to bugs in gerrit comment https://review.openstack.org/71148 | 23:30 |
| fungi | i fixed it in pypi-mirror, and only found 3 total remaining | 23:30 |
| clarkb | fungi: that isn't too bad | 23:30 |
| *** mriedem has joined #openstack-infra | 23:30 | |
| fungi | but there are a handful of others not in global requirements for non-integrated projects too | 23:30 |
| fungi | we may discover more of those once it's allowed to be broken longer | 23:31 |
| fungi | someone might want to draft a quick "we're about to open the gates of the abyss and let the demons loose" e-mail to the dev list | 23:31 |
| *** flaper87 is now known as flaper87|afk | 23:31 | |
| lifeless | anteaya: devananda: https://review.openstack.org/#/c/71046/2 has merged | 23:31 |
| zaro | pleia2: done | 23:31 |
| lifeless | this should make neutron isolated happy with no file injection | 23:31 |
| fungi | because i'm quite positive all integrated gating will be flat out broken while everyone gets caught up | 23:31 |
| lifeless | assuming no further hidden bugs | 23:32 |
| clarkb | fungi: that is what I am thinking, something along the lines of everyone using python setup.py test should s/testr-args=/-t / and be prepared to fix your external project dependencies | 23:32 |
| clarkb | fungi: we can continue to do the patch bomb thing but I think projects have become to reliant on that | 23:32 |
| clarkb | mordred shouldn't have to write 60 patches every cycle to fix silly project things | 23:32 |
| jog0 | clarkb: if you have some time today https://review.openstack.org/#/q/status:open+branch:master+topic:graph-gate,n,z | 23:32 |
| jog0 | if not, ignore me | 23:32 |
| anteaya | lifeless: thank you, yes markmcclain is testing a patch to see how happy neutron is now | 23:33 |
| fungi | clarkb: the last round of those i did for some tox.ini fixup, half of them merged and the other half got randomly downvoted for various reasons and eventually abandoned because i didn't have time to address them all | 23:33 |
| clarkb | mordred: or do you prefer to do patch bombs? | 23:33 |
| *** hogepodge has joined #openstack-infra | 23:33 | |
| mordred | clarkb: I prefer to not do patch bombs | 23:33 |
| clarkb | fungi: right, and imo infra exists to run the tests. Nova being incompatible with tox pip and virtualenv isn't an infra problem | 23:34 |
| mordred | clarkb: I consider icehouse pretty successful because we've done almost no dev on pbr! | 23:34 |
| anteaya | translations from transifex should be fairly benign: https://review.openstack.org/#/c/66501/ | 23:34 |
| clarkb | fungi: we can still run the tests, they just fail :) | 23:34 |
| fungi | mordred: stale pbr! the best kind! | 23:34 |
| mordred | nice | 23:34 |
| *** atiwari has quit IRC | 23:35 | |
| *** rfolco has quit IRC | 23:36 | |
| *** yamahata has quit IRC | 23:38 | |
| *** thuc_ has quit IRC | 23:38 | |
| *** yamahata has joined #openstack-infra | 23:40 | |
| *** andreaf has joined #openstack-infra | 23:42 | |
| clarkb | I am drafting a ml thread starter now | 23:42 |
| mordred | clarkb: I just +2'd https://review.openstack.org/#/c/66072/ - but did not +A | 23:42 |
| mordred | clarkb: becase I'm going to sleep - and I thought that would be rude, but I think we're all generally in favor of it | 23:43 |
| mordred | fungi: ^^ | 23:43 |
| clarkb | mordred: np before you go | 23:43 |
| mordred | clarkb: yah? | 23:43 |
| clarkb | mordred: do you think we should set a date for unpinning and give projects a list of things to do? | 23:43 |
| *** andreaf has quit IRC | 23:43 | |
| clarkb | or do we need to be nicer and unpin after these thigns have been done? | 23:43 |
| anteaya | clarkb: set a date | 23:43 |
| mordred | yeah | 23:44 |
| anteaya | otherwise you will wait for ever | 23:44 |
| mordred | but be nice about it | 23:44 |
| anteaya | a nice date | 23:44 |
| fungi | set a date mainly so that sdague knows to be on vacation | 23:44 |
| anteaya | ha ha ha | 23:44 |
| anteaya | or at least wake up after you, fungi | 23:44 |
| fungi | i don't think he's capable of sleeping past 5am | 23:44 |
| anteaya | salv-orlando thought he lived in the middle of teh atlantic | 23:45 |
| anteaya | based upon his start work time | 23:45 |
| fungi | some day, i will live in the middle of the atlantic, but even then he'll probably still be getting up earlier than me | 23:46 |
| *** thuc has joined #openstack-infra | 23:46 | |
| anteaya | probably | 23:46 |
| anteaya | but you can tell us good stories | 23:46 |
| anteaya | of living on the waves | 23:46 |
| StevenK | The Debian LDAP database contains lat/long for each developer -- years ago, a bunch of developers got mailed saying that the co-ordinates had been plotted, and they appeared to live in an ocean. | 23:47 |
| StevenK | One of my workmates at the time was one of the lucky few to get such an e-mail, and I told him he should replied that he was living on a boat in the middle of the Pacific | 23:48 |
| *** atiwari has joined #openstack-infra | 23:48 | |
| *** thuc_ has joined #openstack-infra | 23:53 | |
| anteaya | I could live on a boat too | 23:53 |
| anteaya | Atlantic, Pacific, not picky just water | 23:53 |
| StevenK | anteaya: The Med is now feeling left out :-P | 23:54 |
| *** markmcclain has joined #openstack-infra | 23:54 | |
| *** yamahata has quit IRC | 23:54 | |
| anteaya | yeah, I seem to manage that | 23:54 |
| anteaya | leaving someone/thing out | 23:54 |
| clarkb | gah the '{posargs}' replaces the quotes | 23:54 |
| anteaya | not sure I could do the Med | 23:54 |
| clarkb | how are we supposed to use this thing? | 23:54 |
| anteaya | too many problems from people trying to flee from the south | 23:55 |
| clarkb | actually tox is on crack | 23:55 |
| fungi | clarkb: that sounds mildly unsafe. though maybe i'm missing relevant context that makes it not a shell vulnerability waiting to happen | 23:55 |
| *** thuc has quit IRC | 23:56 | |
| clarkb | fungi: it means that -t {posargs} won't do what we want it | 23:56 |
| *** whoops has quit IRC | 23:56 | |
| clarkb | we will get -t foo bar baz instead of -t 'foo bar baz' | 23:56 |
| anteaya | clarkb: careful, tox might become mayor ford and refuse to leave office after stripped of all power | 23:57 |
| anteaya | you don't know any canadians, but I will put money on you know who mayor ford is | 23:57 |
| *** branen has joined #openstack-infra | 23:57 | |
| *** weshay has joined #openstack-infra | 23:58 | |
| clarkb | ugh | 23:58 |
| clarkb | I don't know how we can work around this | 23:58 |
| clarkb | I am going to comment on the bug upstream | 23:59 |
| clarkb | but this is seriously insane | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
