| jogo | clarkb: that is why I was hesitant to refactor that code until I knew the logic at least worked | 00:01 |
|---|---|---|
| jogo | as it took me a few revs to get it right too | 00:01 |
| clarkb | jogo: :) it has really helped my understanding of a lot of the stuff that goes on there though | 00:01 |
| jogo | err working not right per se | 00:01 |
| clarkb | and that should get us arbitrary subnode support and ssh known hosts properly everywhere | 00:02 |
| clarkb | so fingers crossed this actually works this time | 00:02 |
| jogo | clarkb: oh sudo vs the tee trick | 00:02 |
| jogo | I thought the tee thing was pretty nifty | 00:02 |
| clarkb | jogo: ya this runs after we chown that dir, which means I have to sudo stuff there | 00:03 |
| clarkb | I think I will try moving it to before the chown at some point but that makes the refactor even bigger | 00:03 |
| *** Guest60909 has quit IRC | 00:03 | |
| *** gokrokve has joined #openstack-infra | 00:04 | |
| *** unicell has quit IRC | 00:04 | |
| jogo | clarkb: looks like we hit nodepool quota | 00:05 |
| jogo | so it will take time to get feedback on those changes now | 00:05 |
| *** cody-somerville has quit IRC | 00:06 | |
| clarkb | jogo: its running now | 00:06 |
| clarkb | https://jenkins06.openstack.org/job/check-tempest-dsvm-aiopcpu/18/console | 00:06 |
| mtreinish | fungi: it looks like rev2 on https://review.openstack.org/#/c/135121/ passed jenkins | 00:06 |
| *** achanda has joined #openstack-infra | 00:08 | |
| fungi | mtreinish: it feels like that's not the right way to address that conflict, but i may be overthinking it. that same conffile is used by both the server and worker classes, and therefire the dburi has to be passed in through both of them. there ought to be a way to just do that in init.pp instead | 00:08 |
| *** dims_ has quit IRC | 00:08 | |
| fungi | except i guess we'd need to instantiate tboth the base class and worker or server classes to be able to do that? | 00:08 |
| mtreinish | fungi: the conf file is not used by the server.pp right now | 00:09 |
| fungi | clarkb: ^ ? | 00:09 |
| fungi | mtreinish: oh, just the dburi is i guess? | 00:09 |
| mtreinish | it just inline the uri on the db manage cli command | 00:09 |
| fungi | got it | 00:09 |
| fungi | eh, that's probably okay then | 00:09 |
| *** chuckC_ has quit IRC | 00:10 | |
| mtreinish | fungi: it was a bit more difficult to do that on the worker side because I used oslo config, so I'd have to play some games with overriding the config value with something I passed in via argparse | 00:10 |
| mtreinish | it's doable, but I was holding off on getting things working before I did that cleanup | 00:10 |
| *** thedodd has quit IRC | 00:11 | |
| jogo | clarkb: nice | 00:12 |
| *** dangers is now known as dangers_away | 00:12 | |
| jogo | clarkb: some of the patches lower down the stack can't accept rechecks yet | 00:12 |
| *** david-lyle is now known as david-lyle_afk | 00:13 | |
| clarkb | jogo: I don't think that matters. we know the first patch makes the ssh stuff fail reliably | 00:13 |
| clarkb | jogo: and the other three are being tested now | 00:13 |
| clarkb | (at least not for the immediate future, I am more interested in specific results right now than jenkins +1s) | 00:13 |
| *** ChuckC has joined #openstack-infra | 00:13 | |
| jogo | clarkb: yeah its not a big deal | 00:15 |
| *** otter768 has joined #openstack-infra | 00:16 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add non voting rally performance job to zaqar project https://review.openstack.org/135132 | 00:17 |
| *** atiwari has quit IRC | 00:18 | |
| *** lttrl has quit IRC | 00:19 | |
| *** cody-somerville has joined #openstack-infra | 00:19 | |
| clarkb | jogo: woot tempest is doing stuff on that change | 00:20 |
| mtreinish | clarkb: I thought we got tempest passing in Paris? | 00:21 |
| mtreinish | I haven't really been following today | 00:21 |
| *** ryanpetrello has joined #openstack-infra | 00:21 | |
| clarkb | jogo: /etc/ssh/ssh_known_hosts looks good on those nodes too | 00:21 |
| clarkb | mtreinish: we did. but apparently there was a gap in ssh known hosts and nova resize/migrations | 00:22 |
| clarkb | mtreinish: we were not properly accepting host keys so ssh would fail in some cases. jogo exposed that by always forcing those ops onto the other node | 00:22 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Enable Ironic periodic-stable jobs https://review.openstack.org/135133 | 00:22 |
| jogo | clarkb: nice! | 00:23 |
| mtreinish | clarkb: ah, ok. It's always something | 00:23 |
| *** adalbas has quit IRC | 00:24 | |
| jogo | mtreinish: yup | 00:25 |
| *** sabeen has joined #openstack-infra | 00:25 | |
| *** ChuckC has quit IRC | 00:25 | |
| jogo | clarkb: woot resize passed on that patch | 00:26 |
| jogo | Failed: 0 | 00:26 |
| clarkb | WOOT | 00:27 |
| clarkb | so now I need to rebase my thing on that change | 00:27 |
| *** ZZelle_ has quit IRC | 00:27 | |
| anteaya | would someone have the time to disable 10386 odl-jenkins OpenDaylight CI odl-openstack@opendaylight.org | 00:28 |
| anteaya | please? | 00:28 |
| mtreinish | jogo: have we tried the block migration test yet? http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/test_live_block_migration.py | 00:28 |
| fungi | anteaya: i can do it now | 00:29 |
| anteaya | fungi: thanks | 00:29 |
| jogo | mtreinish: about to do another run of tempest-full | 00:29 |
| jogo | https://review.openstack.org/#/c/134639/14 | 00:29 |
| fungi | anteaya: done | 00:29 |
| anteaya | fungi: thank you | 00:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests https://review.openstack.org/131604 | 00:29 |
| clarkb | jogo: ^ rebased my change going to check experimental that now | 00:30 |
| clarkb | but that gets us on the path of nova multi node alongside your tempest full stuff | 00:30 |
| clarkb | all very exciting | 00:30 |
| jogo | clarkb: very exciting yeah | 00:30 |
| jogo | mtreinish: http://logs.openstack.org/39/134639/7/experimental/check-tempest-dsvm-aiopcpu/31adb1f/logs/testr_results.html.gz | 00:30 |
| jogo | hmm we are skipping pt272.2: tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestXML.test_live_block_migration[gate] | 00:31 |
| *** funzo has quit IRC | 00:31 | |
| * jogo wonders why | 00:31 | |
| *** whoops has quit IRC | 00:31 | |
| mtreinish | jogo: the tempest flag probably defaults false | 00:32 |
| mtreinish | and devstack probably doesn't understand how to change that | 00:32 |
| fungi | clarkb: any objections to 135121? i have time for one more attempt to build the subunit worker tonight if that merges soon | 00:32 |
| * clarkb looks | 00:32 | |
| mtreinish | jogo: although I'm not sure the compute-feature-enable.live_migration flag should default false | 00:33 |
| mtreinish | since there is also a multiple host check which will raise a skip | 00:33 |
| clarkb | fungi: mtreinish what does that get us? | 00:33 |
| *** Ark has joined #openstack-infra | 00:33 | |
| fungi | clarkb: avoids puppet complaining about a duplicate declaration of /etc/logstash/subunit2sql.conf | 00:34 |
| mtreinish | clarkb: it should unbreak the manifest, the issue is puppet complains about duplicate definitions for everything that doesn't have a suffix var in the name | 00:34 |
| jogo | mtreinish: so should I push a patch to remove that conf flag? | 00:34 |
| *** Ark is now known as Guest25719 | 00:34 | |
| jogo | mtreinish: or turn it on by default? | 00:34 |
| mtreinish | jogo: no we still need the flag, because some hypervisors don't support live migration (I think) | 00:34 |
| jogo | mtreinish: so turn it on by default then? | 00:34 |
| fungi | clarkb: because we instantiate subunit2sql::worker four times in the manifest being applied to teh machine | 00:34 |
| mtreinish | but yeah we can change the default to true (if things blow up we'll know why) | 00:34 |
| jogo | mtreinish: yup | 00:34 |
| mtreinish | clarkb: I guess I didn't have to move everything over to init.pp but it seemed liked the better way | 00:35 |
| * jogo is super excited for multinode | 00:35 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:35 | |
| kevinbenton | has anyone run into an issue where devstack will not exit from ./stack.sh when it’s being invoked via ssh? | 00:37 |
| jogo | mtreinish: what is the difference between live_migration and block_live_migration? | 00:37 |
| fungi | kevinbenton: sounds like you probably have an open file descriptor? | 00:37 |
| fungi | kevinbenton: are you trying to start devstack over ssh and then disconnect and leave it running? | 00:37 |
| kevinbenton | fungi: yes, this is for a 3rd party CI system | 00:38 |
| fungi | kevinbenton: you probably need to redirect stdin, stdout and stderr of whatever command you're backgrounding | 00:39 |
| *** signed8bit_ZZZzz is now known as signed8bit | 00:39 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Change rally job to use fake drivers in cinder https://review.openstack.org/135137 | 00:39 |
| mtreinish | jogo: dunno | 00:40 |
| mtreinish | those tests pre-date me working on tempest I think | 00:40 |
| jogo | ttx: I think one is cinder related | 00:40 |
| fungi | kevinbenton: and possibly fds above 2 as well depending on what it does | 00:40 |
| kevinbenton | fungi: hmm. well it’s just “ssh ubuntu@$NODE_IP ‘~/devstack/stack.sh’” | 00:40 |
| *** ryanpetrello has quit IRC | 00:40 | |
| jogo | doh, mtreinish ^ | 00:40 |
| kevinbenton | fungi: and it worked fine when it was running on 12.10, this only showed up when i changed to 14.04 | 00:40 |
| mtreinish | jogo: well there is one test which requires cinder and iscsi | 00:40 |
| fungi | kevinbenton: try "ssh ubuntu@$NODE_IP? '~/devstack/stack.sh </dev/null >/dev/null 2>&1 &'" | 00:40 |
| mtreinish | and another which says it doesn't | 00:40 |
| kevinbenton | fungi: will do | 00:40 |
| mtreinish | but they both say they block migrate | 00:40 |
| mtreinish | jogo: let me dig up the client api call | 00:41 |
| *** unicell has joined #openstack-infra | 00:41 | |
| *** emagana has joined #openstack-infra | 00:41 | |
| mtreinish | jogo: it looks like it's just live migration: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/services/compute/json/servers_client.py#n405 | 00:42 |
| kevinbenton | fungi: oh, but that immediately returns | 00:42 |
| mtreinish | and one test case does it with an attached volume | 00:42 |
| kevinbenton | fungi: i need to actually wait until stack.sh finishes | 00:42 |
| jogo | mtreinish: are the servers created the same way though? | 00:42 |
| fungi | kevinbenton: in that case, everything stack.sh does needs to properly close out or redirect its file descriptors. will likely take some experimentation to figure out what's leaving some open but not disassociating from the calling process | 00:43 |
| mtreinish | jogo: yep, looking at the test code both are created the same way | 00:43 |
| *** ChuckC has joined #openstack-infra | 00:43 | |
| kevinbenton | fungi: ok, doesn’t infra do something similar to setup its test nodes? | 00:44 |
| fungi | kevinbenton: nope. we only ever start and use devstack once, then discard the machine and grab a fresh one | 00:44 |
| jogo | mtreinish: one attaches a volume to the instance | 00:44 |
| kevinbenton | fungi: yeah, that’s what i’m doing here too | 00:44 |
| jogo | block_migrate_for_live_migration does | 00:45 |
| jogo | we should eventually turn that on too | 00:45 |
| kevinbenton | fungi: this script that sets up devstack on a remote machine does it to a fresh image each time | 00:45 |
| jogo | but lets get one working first | 00:45 |
| jogo | will post a second patch to turn that on as a WIP | 00:45 |
| fungi | kevinbenton: currently devstack is started as a child of a jenkins slave agent process, and does it on the local machine where the job is being run | 00:45 |
| *** chuckC_ has joined #openstack-infra | 00:45 | |
| *** bswartz has joined #openstack-infra | 00:45 | |
| fungi | clarkb: though perhaps that ^ changes for subnodes in a multi-node devstack now? | 00:46 |
| openstackgerrit | Merged openstack-infra/system-config: Move worker pieces that are only need once to init.pp https://review.openstack.org/135121 | 00:46 |
| kevinbenton | fungi: ok, this seems to only happen when stack.sh is triggered via ssh | 00:46 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Clarify changes needed for new puppet repo https://review.openstack.org/134393 | 00:46 |
| *** zz_dimtruck is now known as dimtruck | 00:46 | |
| clarkb | fungi: it does, we run it over ssh | 00:48 |
| clarkb | https://jenkins06.openstack.org/job/check-tempest-dsvm-aiopcpu/18/console | 00:48 |
| fungi | clarkb: do we hold the ssh socket open for the duration of the job? | 00:48 |
| clarkb | kevinbenton: fungi ^ if you search for stack.sh there you will see where it happens | 00:48 |
| mtreinish | jogo: the only difference looks like one attaches a volume before it runs live migrate on the server | 00:48 |
| clarkb | fungi: no, we only run stack.sh during a small portion of the job | 00:48 |
| mtreinish | jogo: d'oh you just said that | 00:48 |
| fungi | clarkb: oh, so at least for the devstack configurations on our subnodes, stack.sh is allowing the ssh connection to close once it completes | 00:48 |
| *** bradjones has quit IRC | 00:48 | |
| clarkb | yes | 00:48 |
| *** amitgandhinz has joined #openstack-infra | 00:48 | |
| clarkb | then tempest runs and talks to the resulting cloud | 00:48 |
| *** ChuckC has quit IRC | 00:48 | |
| *** bradjones has joined #openstack-infra | 00:48 | |
| *** ChuckC has joined #openstack-infra | 00:49 | |
| fungi | kevinbenton: so that suggests that in at least some devstack configurations (on ubuntu trusty) it's not hanging the callingn ssh socket after completion | 00:49 |
| *** dims has joined #openstack-infra | 00:49 | |
| kevinbenton | fungi: ok. i’ll have to figure out what is special here | 00:50 |
| clarkb | sudo -H -u stack stdbuf -oL -eL ./stack.sh > /dev/null is how we run it there | 00:51 |
| *** stevemar has quit IRC | 00:52 | |
| clarkb | maybe the stdbuf is important? | 00:52 |
| kevinbenton | clarkb: i saw that. what do the -oL and -eL and stdbuf do? | 00:52 |
| clarkb | kevinbenton: it says line buffer stdout and stderr | 00:52 |
| kevinbenton | clarkb: oh i see | 00:52 |
| *** stevemar has joined #openstack-infra | 00:52 | |
| kevinbenton | stdbuf is a command, not a magic sudo option i couldn’t find | 00:52 |
| clarkb | yup | 00:53 |
| *** amitgandhinz has quit IRC | 00:53 | |
| kevinbenton | let me try with that | 00:53 |
| clarkb | so we are sudoing stdbuf which then runs stack.sh | 00:53 |
| clarkb | with line buffering | 00:53 |
| kevinbenton | this is annoying to troubleshoot. have to wait ~7 minutes to see if what i changed worked… | 00:54 |
| jogo | mtreinish: https://review.openstack.org/135141 | 00:54 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: Add required puppet module files and .gitreview https://review.openstack.org/134373 | 00:55 |
| *** gokrokve has quit IRC | 00:56 | |
| *** gokrokve has joined #openstack-infra | 00:57 | |
| *** gokrokve has quit IRC | 00:57 | |
| *** bhuvan has joined #openstack-infra | 00:58 | |
| clarkb | jogo: I would like jeblair's input on the use private IPs everywhere change. He original had concerns with that a while back but I think those are no longer issues | 00:59 |
| *** dannywil_ has quit IRC | 01:00 | |
| kevinbenton | clarkb: unfortunately that didn’t do it | 01:00 |
| clarkb | jogo: I much prefer consistency and would prefer that we do that instead of putting some things on private like gre and other things on public like nova | 01:00 |
| clarkb | my change looks happy now too | 01:00 |
| clarkb | jogo: you should probably check experimental the tempest full change now too? | 01:00 |
| jogo | clarkb: yeah I am on in rush to actually land these patches | 01:00 |
| jogo | clarkb: one step ahead of you already running | 01:01 |
| jogo | https://jenkins05.openstack.org/job/check-tempest-dsvm-aiopcpu/4/console | 01:01 |
| clarkb | https://jenkins06.openstack.org/job/check-tempest-dsvm-aiopcpu/19/console is nova multi host | 01:01 |
| jogo | clarkb: cool if your multi host patch works I will rebase my tempest-full test patch on top of it | 01:02 |
| kevinbenton | clarkb, fungi: it looks like this process is hanging around ubuntu 3023 3018 0 00:54 ? 00:00:00 python /home/ubuntu/devstack/tools/outfilter.py -v -o /opt/stack/logs/devstacklog.txt.2014-11-18-005400 | 01:02 |
| uvirtbot | Launchpad bug 3023 in launchpad "team emails do not set headers right" [Medium,Fix released] https://launchpad.net/bugs/3023 | 01:02 |
| kevinbenton | when i killed that, stack.sh exited as expected | 01:02 |
| jogo | clarkb: the big test for nova will come tomorrow or later though: https://review.openstack.org/#/c/135141/ | 01:04 |
| jogo | mtreinish: we will need another tempest test for doing live migration for boot from volume instances btw | 01:05 |
| *** yamahata has joined #openstack-infra | 01:06 | |
| clarkb | jogo: https://jenkins06.openstack.org/job/check-tempest-dsvm-aiopcpu/19/console succeeded, but don't rebase tempest full until your job testing that is finished running | 01:06 |
| clarkb | if you rebase and push it will stop that job | 01:06 |
| mtreinish | jogo: well you can add it after we get multinode working | 01:07 |
| *** yamahata has quit IRC | 01:07 | |
| *** yamahata has joined #openstack-infra | 01:07 | |
| *** wenlock has quit IRC | 01:08 | |
| *** cody-somerville has quit IRC | 01:08 | |
| *** spzala has joined #openstack-infra | 01:09 | |
| *** yjiang5 is now known as yjiang5_away | 01:09 | |
| *** jaypipes has quit IRC | 01:11 | |
| *** salv-orlando has quit IRC | 01:14 | |
| *** stevemar has quit IRC | 01:14 | |
| asselin | anyone seen this? Parameter name failed on Database_user[root@]: Invalid database user root@ at /etc/puppet/modules/mysql/manifests/server/account_security.pp:8 | 01:15 |
| clarkb | asselin: ya it means that your fqdn fact is empty | 01:16 |
| asselin | clarkb, thanks | 01:16 |
| *** cody-somerville has joined #openstack-infra | 01:16 | |
| *** cody-somerville has quit IRC | 01:16 | |
| *** cody-somerville has joined #openstack-infra | 01:16 | |
| clarkb | you need to manipulate /etc/hosts to make it have an fqdn longer than hostname. | 01:16 |
| fungi | mtreinish: don't hit me, but Could not find dependency Class[Logstash::Indexer] for File[/etc/logstash/jenkins-subunit-worker-A.yaml] at /opt/system-config/production/modules/subunit2sql/manifests/worker.pp:42 | 01:18 |
| fungi | mtreinish: probably needs an 'include logstash::indexer' line? | 01:18 |
| mtreinish | fungi: sigh, yep, looks like it | 01:18 |
| *** MaxV has quit IRC | 01:18 | |
| mtreinish | one more patch... | 01:18 |
| fungi | mtreinish: i'm going to pretend not to work for a while, but can test again tomorrow morning | 01:19 |
| *** salv-orlando has joined #openstack-infra | 01:19 | |
| jogo | clarkb: agreed | 01:19 |
| jogo | mtreinish: right, live migration testing is the feature I am most excited for nova wise | 01:19 |
| mtreinish | fungi: ok, I can bug clarkb then :) | 01:19 |
| *** banix has quit IRC | 01:19 | |
| clarkb | mtreinish: lol I am so jetlagged | 01:19 |
| *** ryanpetrello has joined #openstack-infra | 01:19 | |
| clarkb | barely awake at this point | 01:19 |
| fungi | poor clarkb thinks it's 2am | 01:19 |
| clarkb | I am going to try to make it to 8pm tonight | 01:20 |
| fungi | good luck... you still have a couple hours to go in that case | 01:20 |
| clarkb | jogo: tempest full just succeeded | 01:21 |
| clarkb | jogo: with nova net, still waiting on neutron | 01:21 |
| *** hogepodge has quit IRC | 01:21 | |
| *** amcrn has quit IRC | 01:21 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/elastic-recheck: Add query for test_baremetal_server_ops timeout https://review.openstack.org/134738 | 01:22 |
| *** emagana has quit IRC | 01:24 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add missing logstash import https://review.openstack.org/135145 | 01:24 |
| mtreinish | clarkb, fungi: ^^^ that should fix it | 01:25 |
| mtreinish | right? | 01:25 |
| mtreinish | clarkb: heh, did you just get back home today? | 01:25 |
| clarkb | mtreinish: yesterday | 01:26 |
| clarkb | I then proceeded to get two 4 hour chunks of sleep and was awake by about 4am | 01:26 |
| clarkb | jogo: neutron failed | 01:26 |
| mtreinish | clarkb: ah, ok yeah that was pretty much Sat. for me | 01:27 |
| *** signed8bit has quit IRC | 01:27 | |
| mtreinish | I got back on Fri | 01:27 |
| *** spzala has quit IRC | 01:28 | |
| *** spzala has joined #openstack-infra | 01:28 | |
| *** stevemar has joined #openstack-infra | 01:28 | |
| clarkb | it was a good week | 01:29 |
| *** jyuso has joined #openstack-infra | 01:30 | |
| *** achanda has quit IRC | 01:31 | |
| kevinbenton | an amazingly hacky fix is to kill the python process after stack.sh and the ssh process will then be able to exit | 01:31 |
| kevinbenton | ssh ubuntu@$NODE_IP “~/devstack/stack.sh && kill \$(ps -ef | grep outfilter.py | grep -v grep | awk '{ print \$2 }')" | 01:31 |
| clarkb | kevinbenton: I don't recognize that process. some of the devstack devs might though | 01:31 |
| *** achanda has joined #openstack-infra | 01:32 | |
| kevinbenton | dtroyer: are you around? ^^ | 01:33 |
| stevebaker | sdague, clarkb: I've realised why test_server_cfn_init is failing in non-heat-slow jobs. They use config drive, so user_data is executed before networking is fully up. This is not an issue when polling the nova metadata service | 01:33 |
| mtreinish | stevebaker: oh, did that patch with the loop never land? Or did we end up abandoning it? (or was it on a different test) | 01:35 |
| stevebaker | mtreinish: different test, which is designed to run in the main jobs | 01:35 |
| mtreinish | stevebaker: ah, ok | 01:35 |
| mtreinish | it still seems like something we'd want to be tolerant of in the general case though | 01:35 |
| dtroyer | kevinbenton: I'm in & out | 01:36 |
| mtreinish | because it's not like there's a config flag to specify whether it's config drive or not | 01:36 |
| *** achanda has quit IRC | 01:36 | |
| kevinbenton | dtroyer: just wanted to know if you knew of anything that would block outfilter.py from exiting when stack.sh is executed over ssh | 01:36 |
| stevebaker | mtreinish: I remember, I was attempting to move a different test out of heat-slow and hit this issue https://review.openstack.org/#/c/110507/5 | 01:36 |
| mtreinish | stevebaker: ah, ok yeah that's it | 01:37 |
| *** Ryan_Lane has quit IRC | 01:37 | |
| *** rfolco has quit IRC | 01:37 | |
| *** Ryan_Lane has joined #openstack-infra | 01:37 | |
| stevebaker | mtreinish: I suppose the tests should JustWork whether config drive or not, so there needs to be a retry loop which hits some URL until success, then continue with the other network operations | 01:38 |
| dtroyer | kevinbenton: I'd start by suspecting something connected to the loggin and redirection stack.sh does, I'm not certain how it will behave if there is no tty. | 01:38 |
| mtreinish | stevebaker: yeah, that's what I was getting at | 01:38 |
| dtroyer | can you try the ssh with -t? | 01:38 |
| kevinbenton | dtroyer: yeah | 01:39 |
| *** hogepodge has joined #openstack-infra | 01:39 | |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/134763 | 01:40 |
| openstackgerrit | Merged openstack-infra/project-config: Grouping the two projects Groups portal in storyboard https://review.openstack.org/134330 | 01:40 |
| kevinbenton | dtroyer: whatever it is worked fine on 12.10 but broke on my change to 14.04 | 01:40 |
| *** banix has joined #openstack-infra | 01:41 | |
| dtroyer | kevinbenton: hmmm…that suggests not a tty problem. when you changed to 14.04 you had roughly the same devstack checkout? | 01:41 |
| *** banix has quit IRC | 01:42 | |
| *** MarkAtwood has quit IRC | 01:42 | |
| kevinbenton | dtroyer: yeah | 01:42 |
| kevinbenton | dtroyer: a recent change pushed devstack to use mariadb, which required more hacking to get to work on 12.10 so i figured i would change images today | 01:42 |
| *** teran has quit IRC | 01:43 | |
| kevinbenton | dtroyer: was working fine with 12.10 as of early yesterday | 01:43 |
| *** bhuvan has quit IRC | 01:43 | |
| dtroyer | kevinbenton: can you give me a command like or something to try to reproduce it? | 01:43 |
| kevinbenton | ssh -i ~/keys/openstackCI.key ubuntu@$NODE_IP “~/devstack/stack.sh” | 01:44 |
| *** iain-king has joined #openstack-infra | 01:44 | |
| kevinbenton | dtroyer: that is run from the system controlling the test nodes | 01:45 |
| dtroyer | kevinbenton: ok, simple enough, and I'm wondering why I haven't done that before ;) | 01:46 |
| dtroyer | so it hangs at the noremal end of stack.sh? (I'm running one on trusty now) | 01:46 |
| kevinbenton | dtroyer: yeah | 01:47 |
| kevinbenton | dtroyer: for some reason outfilter.py doesn’t exit, if i do “stack.sh && kill $OUTFILTER_PID” it then exits | 01:48 |
| *** teran has joined #openstack-infra | 01:48 | |
| kevinbenton | dtroyer: so i think stack.sh actually completes, but outfilter is stuck | 01:48 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add oslo.context to projects.txt https://review.openstack.org/135147 | 01:50 |
| kevinbenton | ssh -i $DEVSTACK_SSH_KEY ubuntu@$DEVSTACK_IP "~/devstack/stack.sh && kill \$(ps -ef | grep outfilter.py | grep -v grep | awk '{ print \$2 }')" | 01:50 |
| kevinbenton | dtroyer: ^^ that’s my temporary workaround | 01:50 |
| kevinbenton | dtroyer: i have to step away for a bit, let me know if you need any more info from me | 01:51 |
| mmedvede | kevinbenton: not sure if relevant, but I did run devstack through ssh like this: ssh $USER@$ip -t -t "bash -l -c 'cd devstack && ./stack.sh'" | 01:51 |
| *** rfolco has joined #openstack-infra | 01:52 | |
| kevinbenton | mmedvede: so perhaps the tty is the issue | 01:52 |
| *** tkelsey has joined #openstack-infra | 01:53 | |
| *** banix has joined #openstack-infra | 01:53 | |
| dtroyer | kevinbenton: ok, … fwiw, my run just hung, so repro here | 01:53 |
| *** banix has quit IRC | 01:54 | |
| kevinbenton | dtroyer: ok. i am doing one more run to check if the ‘-t’ flag helps | 01:54 |
| *** bdpayne has quit IRC | 01:55 | |
| mmedvede | kevinbenton: note that one vs two -t flags is different | 01:55 |
| jogo | clarkb: will look in a bit | 01:56 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for test_baremetal_server_ops timeout https://review.openstack.org/134738 | 01:56 |
| *** julim has joined #openstack-infra | 01:56 | |
| *** dims has quit IRC | 01:58 | |
| *** nosnos has joined #openstack-infra | 01:59 | |
| *** tkelsey has quit IRC | 01:59 | |
| *** mriedem has joined #openstack-infra | 02:03 | |
| *** shashankhegde has quit IRC | 02:05 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova/oslo.db pg test bug 1393633 https://review.openstack.org/135149 | 02:08 |
| uvirtbot | Launchpad bug 1393633 in nova "test_postgresql_opportunistically fails with "database "openstack_citest" is being accessed by other users"" [High,Confirmed] https://launchpad.net/bugs/1393633 | 02:08 |
| *** rfolco has quit IRC | 02:08 | |
| *** rfolco has joined #openstack-infra | 02:08 | |
| kevinbenton | mmedvede, dtroyer: a single -t flag seemed to fix it | 02:09 |
| *** rfolco has quit IRC | 02:13 | |
| *** ryanpetrello has quit IRC | 02:14 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova/oslo.db pg test bug 1393633 https://review.openstack.org/135149 | 02:14 |
| uvirtbot | Launchpad bug 1393633 in nova "test_postgresql_opportunistically fails with "database "openstack_citest" is being accessed by other users"" [High,Confirmed] https://launchpad.net/bugs/1393633 | 02:14 |
| *** rfolco has joined #openstack-infra | 02:16 | |
| *** banix has joined #openstack-infra | 02:17 | |
| kevinbenton | mmedvede, dtroyer: two -t flags fixes it as well :-) | 02:20 |
| mmedvede | kevinbenton: yay! | 02:20 |
| *** iain-kin1 has joined #openstack-infra | 02:21 | |
| kevinbenton | mmedvede: no idea why it wouldn’t fail on 12.10 though… | 02:21 |
| *** rfolco has quit IRC | 02:21 | |
| dtroyer | kevinbenton: ok, good. I'm wondering if it a bash issue? when it hangs, the top-evel bash is in a wait4() call for pid -1, which of course doesn't exist | 02:23 |
| mmedvede | kevinbenton: don't know either. Did not work for me on 12.04 | 02:23 |
| *** iain-king has quit IRC | 02:24 | |
| dtroyer | kevinbenton, mmedvede: how do you guys have LOGFILE and VERBOSE set? | 02:24 |
| kevinbenton | LOGFILE=/opt/stack/logs/devstacklog.txt | 02:25 |
| kevinbenton | VERBOSE=True | 02:25 |
| kevinbenton | dtroyer: ^^ | 02:25 |
| mmedvede | dtroyer: I had LOGFILE=$DATA_DIR/logs/stack.log and VERBOSE=True as well | 02:27 |
| dtroyer | ok, thanks, so both have the same logging and fd setup | 02:28 |
| *** gokrokve has joined #openstack-infra | 02:29 | |
| *** Ryan_Lane has quit IRC | 02:31 | |
| *** koolhead17 has joined #openstack-infra | 02:33 | |
| *** mbacchi has quit IRC | 02:37 | |
| dtroyer | I'm not sure how much farther this is worth persuing, at least tonight. What I'm seeing is the top-level bash process getting daemonized and owned by init, while the sshd is still waiting for input... | 02:38 |
| *** Viswanath has joined #openstack-infra | 02:38 | |
| *** oomichi has joined #openstack-infra | 02:40 | |
| *** Viswanath has quit IRC | 02:41 | |
| *** jyuso has quit IRC | 02:41 | |
| *** bhunter71 has joined #openstack-infra | 02:42 | |
| *** armax has quit IRC | 02:43 | |
| *** banix has quit IRC | 02:45 | |
| *** dims has joined #openstack-infra | 02:51 | |
| *** armax has joined #openstack-infra | 02:56 | |
| *** patrickeast has quit IRC | 02:58 | |
| *** chuckC_ has quit IRC | 02:59 | |
| *** gokrokve has quit IRC | 03:00 | |
| *** chuckC_ has joined #openstack-infra | 03:00 | |
| *** gokrokve has joined #openstack-infra | 03:00 | |
| *** bdpayne has joined #openstack-infra | 03:03 | |
| *** armax has quit IRC | 03:07 | |
| *** dims has quit IRC | 03:13 | |
| *** dims has joined #openstack-infra | 03:13 | |
| *** SumitNaiksatam has quit IRC | 03:14 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:14 | |
| *** banix has joined #openstack-infra | 03:17 | |
| *** ivar-laz_ has joined #openstack-infra | 03:21 | |
| *** ivar-laz_ has joined #openstack-infra | 03:22 | |
| *** ivar-laz_ has joined #openstack-infra | 03:23 | |
| *** ivar-lazzaro has quit IRC | 03:23 | |
| *** groknix has joined #openstack-infra | 03:23 | |
| *** ivar-laz_ has quit IRC | 03:23 | |
| *** ivar-lazzaro has joined #openstack-infra | 03:25 | |
| *** spzala has quit IRC | 03:30 | |
| *** armax has joined #openstack-infra | 03:33 | |
| *** nosnos has quit IRC | 03:35 | |
| *** wenlock has joined #openstack-infra | 03:36 | |
| *** stevemar has quit IRC | 03:37 | |
| *** chuckC_ has quit IRC | 03:39 | |
| *** jaypipes has joined #openstack-infra | 03:42 | |
| *** boris-42 has quit IRC | 03:47 | |
| *** bhunter71 has quit IRC | 03:47 | |
| *** koolhead17 has quit IRC | 03:52 | |
| *** gokrokve_ has joined #openstack-infra | 03:52 | |
| *** fandi has quit IRC | 03:52 | |
| *** koolhead17 has joined #openstack-infra | 03:52 | |
| *** gokrokve_ has quit IRC | 03:54 | |
| *** wenlock has quit IRC | 03:55 | |
| *** gokrokve_ has joined #openstack-infra | 03:55 | |
| *** gokrokve has quit IRC | 03:55 | |
| *** stevemar has joined #openstack-infra | 03:56 | |
| *** koolhead17 has quit IRC | 03:56 | |
| *** chandankumar has joined #openstack-infra | 03:58 | |
| *** armax has quit IRC | 03:58 | |
| *** Ryan_Lane has joined #openstack-infra | 03:59 | |
| *** gokrokve_ has quit IRC | 03:59 | |
| *** Ryan_Lane has quit IRC | 04:00 | |
| *** Ryan_Lane has joined #openstack-infra | 04:01 | |
| *** chandankumar has quit IRC | 04:02 | |
| *** armax has joined #openstack-infra | 04:03 | |
| *** mriedem has quit IRC | 04:04 | |
| *** doude has quit IRC | 04:07 | |
| *** ivar-laz_ has joined #openstack-infra | 04:13 | |
| *** ivar-laz_ has joined #openstack-infra | 04:13 | |
| *** subscope has joined #openstack-infra | 04:14 | |
| *** rushiagr_away is now known as rushiagr | 04:14 | |
| *** pcrews has quit IRC | 04:15 | |
| *** ivar-lazzaro has quit IRC | 04:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch to installing subunit2sql from git instead of pip https://review.openstack.org/135161 | 04:17 |
| *** ivar-laz_ has quit IRC | 04:17 | |
| *** banix has quit IRC | 04:18 | |
| *** gokrokve has joined #openstack-infra | 04:22 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch to installing subunit2sql from git instead of pip https://review.openstack.org/135161 | 04:23 |
| *** gokrokve has quit IRC | 04:24 | |
| *** gokrokve has joined #openstack-infra | 04:24 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 04:28 |
| *** gokrokve has quit IRC | 04:29 | |
| jogo | is it possible to set tempest.conf from devstack-gate? | 04:30 |
| *** dimtruck is now known as zz_dimtruck | 04:37 | |
| *** Ryan_Lane has quit IRC | 04:38 | |
| *** armax has quit IRC | 04:40 | |
| *** prad has quit IRC | 04:43 | |
| *** nosnos has joined #openstack-infra | 04:44 | |
| *** BharatK has joined #openstack-infra | 04:46 | |
| *** yfried_ has joined #openstack-infra | 04:51 | |
| *** groknix has quit IRC | 04:55 | |
| *** otherwiseguy has quit IRC | 04:55 | |
| *** otherwiseguy has joined #openstack-infra | 04:58 | |
| *** achanda has joined #openstack-infra | 05:01 | |
| *** banix has joined #openstack-infra | 05:02 | |
| openstackgerrit | Praveen Kumar proposed openstack-infra/jenkins-job-builder: Add perlcritic capability to violations method https://review.openstack.org/134867 | 05:02 |
| *** jerryz has quit IRC | 05:20 | |
| *** SumitNaiksatam has quit IRC | 05:24 | |
| *** gokrokve has joined #openstack-infra | 05:24 | |
| *** oomichi has quit IRC | 05:24 | |
| *** jerryz has joined #openstack-infra | 05:26 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:26 | |
| *** subscope has quit IRC | 05:27 | |
| *** yfried_ has quit IRC | 05:28 | |
| *** gokrokve has quit IRC | 05:28 | |
| *** unicell has quit IRC | 05:30 | |
| *** jerryz has quit IRC | 05:33 | |
| *** chandankumar has joined #openstack-infra | 05:36 | |
| *** miqui has joined #openstack-infra | 05:38 | |
| *** viglesias has quit IRC | 05:42 | |
| openstackgerrit | Steven Weston proposed openstack-infra/infra-specs: Third Party CI Dashboard Spec https://review.openstack.org/135170 | 05:42 |
| *** mdbooth has quit IRC | 05:43 | |
| *** viglesias has joined #openstack-infra | 05:43 | |
| *** banix has quit IRC | 05:43 | |
| *** mdbooth has joined #openstack-infra | 05:48 | |
| *** oomichi has joined #openstack-infra | 05:48 | |
| *** boris-42 has joined #openstack-infra | 05:50 | |
| *** achanda has quit IRC | 05:51 | |
| *** tkelsey has joined #openstack-infra | 05:56 | |
| *** otherwiseguy has quit IRC | 05:57 | |
| *** tkelsey has quit IRC | 06:01 | |
| *** Guest25719 has quit IRC | 06:01 | |
| *** oomichi has quit IRC | 06:02 | |
| *** achanda has joined #openstack-infra | 06:06 | |
| *** viglesias has quit IRC | 06:06 | |
| *** viglesias has joined #openstack-infra | 06:06 | |
| *** oomichi has joined #openstack-infra | 06:08 | |
| *** achanda has quit IRC | 06:08 | |
| *** achanda has joined #openstack-infra | 06:10 | |
| *** renlt has joined #openstack-infra | 06:13 | |
| *** renlt has quit IRC | 06:14 | |
| *** renlt has joined #openstack-infra | 06:14 | |
| *** renlt has quit IRC | 06:16 | |
| *** renlt has joined #openstack-infra | 06:16 | |
| *** renlt has quit IRC | 06:16 | |
| *** renlt has joined #openstack-infra | 06:17 | |
| *** renlt has quit IRC | 06:17 | |
| *** dmellado has joined #openstack-infra | 06:22 | |
| *** shashankhegde has joined #openstack-infra | 06:22 | |
| *** [HeOS] has quit IRC | 06:22 | |
| *** ildikov has quit IRC | 06:22 | |
| *** gokrokve has joined #openstack-infra | 06:24 | |
| *** melwitt has quit IRC | 06:27 | |
| *** gokrokve has quit IRC | 06:29 | |
| *** vigneshvar has joined #openstack-infra | 06:30 | |
| *** shashankhegde has quit IRC | 06:31 | |
| *** bdpayne has quit IRC | 06:32 | |
| *** Longgeek has joined #openstack-infra | 06:34 | |
| *** Ark has joined #openstack-infra | 06:36 | |
| *** Ark is now known as Guest99275 | 06:36 | |
| *** todd_dsm has quit IRC | 06:37 | |
| *** melwitt has joined #openstack-infra | 06:39 | |
| *** melwitt1 has joined #openstack-infra | 06:42 | |
| *** melwitt has quit IRC | 06:44 | |
| *** harlowja is now known as harlowja_away | 06:45 | |
| *** melwitt1 has quit IRC | 06:46 | |
| *** dmellado has quit IRC | 06:55 | |
| *** yfried_ has joined #openstack-infra | 06:56 | |
| *** k4n0 has joined #openstack-infra | 06:57 | |
| *** Guest99275 has quit IRC | 06:58 | |
| *** achanda has quit IRC | 06:59 | |
| *** belmoreira has joined #openstack-infra | 07:05 | |
| *** yfried_ has quit IRC | 07:14 | |
| *** yfried__ has joined #openstack-infra | 07:14 | |
| *** katyafervent_awa is now known as katyafervent | 07:16 | |
| *** amuller has joined #openstack-infra | 07:20 | |
| *** gokrokve has joined #openstack-infra | 07:24 | |
| *** hdd has joined #openstack-infra | 07:28 | |
| *** gokrokve has quit IRC | 07:29 | |
| *** amotoki has joined #openstack-infra | 07:32 | |
| *** dmellado has joined #openstack-infra | 07:34 | |
| *** fandi has joined #openstack-infra | 07:36 | |
| *** denis_makogon has joined #openstack-infra | 07:38 | |
| *** denis_makogon has quit IRC | 07:38 | |
| *** achanda has joined #openstack-infra | 07:42 | |
| *** afazekas has joined #openstack-infra | 07:45 | |
| *** dmellado has quit IRC | 07:51 | |
| *** Longgeek has quit IRC | 07:53 | |
| *** Longgeek has joined #openstack-infra | 07:53 | |
| *** yfried__ is now known as yfried|afk | 07:56 | |
| *** yamamoto_ has joined #openstack-infra | 07:56 | |
| *** HeOS has joined #openstack-infra | 07:56 | |
| *** Longgeek_ has joined #openstack-infra | 07:57 | |
| *** yfried|afk has quit IRC | 08:00 | |
| *** Longgeek has quit IRC | 08:00 | |
| *** achanda has quit IRC | 08:02 | |
| *** katyafervent is now known as katyafervent_awa | 08:06 | |
| *** achuprin_ has quit IRC | 08:07 | |
| *** Ark has joined #openstack-infra | 08:09 | |
| *** ildikov has joined #openstack-infra | 08:09 | |
| *** Longgeek_ has quit IRC | 08:09 | |
| *** Ark is now known as Guest46043 | 08:09 | |
| *** Longgeek has joined #openstack-infra | 08:10 | |
| *** miqui has quit IRC | 08:11 | |
| *** mrmartin has quit IRC | 08:12 | |
| *** otter768 has quit IRC | 08:12 | |
| *** Guest46043 has quit IRC | 08:13 | |
| *** mrmartin has joined #openstack-infra | 08:13 | |
| *** arxcruz has joined #openstack-infra | 08:14 | |
| *** Longgeek_ has joined #openstack-infra | 08:14 | |
| *** yfried|afk has joined #openstack-infra | 08:16 | |
| *** Longgeek has quit IRC | 08:18 | |
| *** gilliard has quit IRC | 08:21 | |
| *** links has joined #openstack-infra | 08:21 | |
| *** gilliard has joined #openstack-infra | 08:21 | |
| *** gokrokve has joined #openstack-infra | 08:24 | |
| *** yamamoto_ has quit IRC | 08:25 | |
| *** gilliard has quit IRC | 08:26 | |
| *** jcoufal has joined #openstack-infra | 08:26 | |
| *** ameade has quit IRC | 08:26 | |
| *** sweston has quit IRC | 08:26 | |
| *** erw has quit IRC | 08:27 | |
| *** ameade_ has joined #openstack-infra | 08:27 | |
| *** teran has quit IRC | 08:27 | |
| *** sweston_ has joined #openstack-infra | 08:27 | |
| *** erw has joined #openstack-infra | 08:27 | |
| *** gokrokve has quit IRC | 08:29 | |
| *** jlibosva has joined #openstack-infra | 08:31 | |
| *** gilliard has joined #openstack-infra | 08:31 | |
| *** koolhead17 has joined #openstack-infra | 08:32 | |
| *** luqas has joined #openstack-infra | 08:34 | |
| *** MaxV has joined #openstack-infra | 08:36 | |
| *** mpaolino has joined #openstack-infra | 08:36 | |
| *** MaxV has quit IRC | 08:39 | |
| *** matt_____ is now known as mattoliverau | 08:39 | |
| *** jawed has joined #openstack-infra | 08:40 | |
| *** mrmartin has quit IRC | 08:40 | |
| *** henditan has joined #openstack-infra | 08:41 | |
| *** nadya_ has joined #openstack-infra | 08:46 | |
| *** doude has joined #openstack-infra | 08:46 | |
| *** ashaeron has joined #openstack-infra | 08:49 | |
| *** dizquierdo has joined #openstack-infra | 08:50 | |
| *** rcarrill` has joined #openstack-infra | 08:52 | |
| *** MaxV has joined #openstack-infra | 08:58 | |
| *** Hal has joined #openstack-infra | 09:02 | |
| *** vigneshvar has quit IRC | 09:02 | |
| *** Hal is now known as Guest40266 | 09:02 | |
| *** derekh has joined #openstack-infra | 09:03 | |
| *** cnesa has joined #openstack-infra | 09:04 | |
| *** yfried|afk is now known as yfried__ | 09:05 | |
| *** jpich has joined #openstack-infra | 09:06 | |
| *** imcsk8 has quit IRC | 09:06 | |
| *** imcsk8 has joined #openstack-infra | 09:06 | |
| *** Guest40266 has quit IRC | 09:07 | |
| *** Hal_ has joined #openstack-infra | 09:07 | |
| *** reed has joined #openstack-infra | 09:09 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:10 | |
| *** dtantsur|afk is now known as dtantsur | 09:10 | |
| *** jcoufal has quit IRC | 09:11 | |
| *** [HeOS] has joined #openstack-infra | 09:11 | |
| *** jcoufal has joined #openstack-infra | 09:13 | |
| *** jcoufal has quit IRC | 09:14 | |
| *** jcoufal has joined #openstack-infra | 09:14 | |
| *** HeOS has quit IRC | 09:14 | |
| *** jedimike has joined #openstack-infra | 09:15 | |
| *** SotK has joined #openstack-infra | 09:16 | |
| *** bo_sh has joined #openstack-infra | 09:17 | |
| *** teran has joined #openstack-infra | 09:17 | |
| *** teran_ has joined #openstack-infra | 09:18 | |
| *** BharatK has quit IRC | 09:20 | |
| *** teran has quit IRC | 09:22 | |
| *** gokrokve has joined #openstack-infra | 09:24 | |
| *** yamamoto_ has joined #openstack-infra | 09:25 | |
| *** hdd has quit IRC | 09:26 | |
| *** gokrokve has quit IRC | 09:29 | |
| *** dmellado has joined #openstack-infra | 09:29 | |
| *** lttrl has joined #openstack-infra | 09:30 | |
| *** dmellado has quit IRC | 09:30 | |
| *** yamamoto_ has quit IRC | 09:30 | |
| *** hdd has joined #openstack-infra | 09:30 | |
| *** dmellado has joined #openstack-infra | 09:31 | |
| *** BharatK has joined #openstack-infra | 09:32 | |
| *** yfried__ is now known as yfried|afk | 09:34 | |
| *** yfried|afk is now known as yfried__ | 09:36 | |
| *** lttrl has quit IRC | 09:36 | |
| *** lttrl has joined #openstack-infra | 09:37 | |
| *** aalekov has quit IRC | 09:38 | |
| *** jamielennox is now known as jamielennox|away | 09:43 | |
| *** dtantsur is now known as dtantsur|afk | 09:45 | |
| *** tlbr has joined #openstack-infra | 09:46 | |
| *** KanagarajM has joined #openstack-infra | 09:47 | |
| *** unicell has joined #openstack-infra | 09:50 | |
| *** amuller_ has joined #openstack-infra | 09:52 | |
| *** amuller has quit IRC | 09:52 | |
| *** vrutkovs has joined #openstack-infra | 09:53 | |
| *** nadya_ has quit IRC | 09:54 | |
| *** yfried__ is now known as yfried|afk | 10:02 | |
| *** rlandy has joined #openstack-infra | 10:02 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: tooz: gate against py{27,34}-mysql https://review.openstack.org/135206 | 10:06 |
| *** pelix has joined #openstack-infra | 10:14 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Create puppet-forge credentials on the pypi slave https://review.openstack.org/134835 | 10:15 |
| *** ilyashakhat has joined #openstack-infra | 10:16 | |
| *** yfried|afk is now known as yfried | 10:20 | |
| *** gokrokve has joined #openstack-infra | 10:24 | |
| *** pelix has quit IRC | 10:24 | |
| *** pelix has joined #openstack-infra | 10:25 | |
| *** henditan has quit IRC | 10:25 | |
| *** BharatK has quit IRC | 10:26 | |
| *** unicell has quit IRC | 10:26 | |
| *** jgallard has joined #openstack-infra | 10:26 | |
| *** gokrokve has quit IRC | 10:29 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 10:32 |
| *** amuller_ has quit IRC | 10:36 | |
| *** ihrachyshka has joined #openstack-infra | 10:37 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 10:38 |
| *** BharatK has joined #openstack-infra | 10:38 | |
| *** stevemar has quit IRC | 10:39 | |
| *** jp_at_hp has joined #openstack-infra | 10:40 | |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| *** vigneshvar has joined #openstack-infra | 10:42 | |
| *** Hal_ has quit IRC | 10:42 | |
| *** iain-kin1 has quit IRC | 10:45 | |
| *** ldnunes has joined #openstack-infra | 10:50 | |
| *** katyafervent_awa has quit IRC | 10:58 | |
| *** tsufiev has quit IRC | 10:58 | |
| *** dteselkin has quit IRC | 10:58 | |
| *** rakhmerov has quit IRC | 10:58 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** Longgeek_ has quit IRC | 11:04 | |
| *** masayukig has quit IRC | 11:04 | |
| *** masayukig has joined #openstack-infra | 11:04 | |
| *** mrmartin has joined #openstack-infra | 11:04 | |
| *** Longgeek has joined #openstack-infra | 11:05 | |
| *** nadya_ has joined #openstack-infra | 11:05 | |
| *** [HeOS] is now known as HeOS | 11:08 | |
| *** jgallard has quit IRC | 11:08 | |
| *** yfried is now known as yfried|afk | 11:10 | |
| *** teran_ has quit IRC | 11:13 | |
| *** jawed has quit IRC | 11:15 | |
| *** ihrachyshka has quit IRC | 11:16 | |
| *** ihrachyshka has joined #openstack-infra | 11:17 | |
| *** bo_sh has quit IRC | 11:17 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 11:18 |
| *** yfried|afk is now known as yfried | 11:18 | |
| *** ihrachyshka has quit IRC | 11:20 | |
| *** pc_m has joined #openstack-infra | 11:23 | |
| *** pc_m has quit IRC | 11:24 | |
| *** gokrokve has joined #openstack-infra | 11:24 | |
| *** pc_m has joined #openstack-infra | 11:25 | |
| *** aysyd has joined #openstack-infra | 11:27 | |
| openstackgerrit | Merged openstack/requirements: Add oslo.context to projects.txt https://review.openstack.org/135147 | 11:28 |
| *** gokrokve has quit IRC | 11:29 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/135233 | 11:29 |
| *** dims has quit IRC | 11:32 | |
| *** dims has joined #openstack-infra | 11:32 | |
| *** amuller_ has joined #openstack-infra | 11:32 | |
| *** bo_sh has joined #openstack-infra | 11:35 | |
| *** chandan_kumar has joined #openstack-infra | 11:37 | |
| *** bradjones_ has joined #openstack-infra | 11:38 | |
| *** bradjones has quit IRC | 11:39 | |
| *** bradjones_ is now known as bradjones | 11:39 | |
| *** bradjones has joined #openstack-infra | 11:39 | |
| *** chandankumar has quit IRC | 11:40 | |
| *** nuritv has joined #openstack-infra | 11:41 | |
| *** mpaolino has quit IRC | 11:53 | |
| *** Hal has joined #openstack-infra | 12:00 | |
| *** Hal is now known as Guest67947 | 12:00 | |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| *** BharatK has left #openstack-infra | 12:06 | |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/135233 | 12:07 |
| *** yamamoto has quit IRC | 12:09 | |
| *** baoli has quit IRC | 12:13 | |
| *** KanagarajM has quit IRC | 12:13 | |
| *** MaxV has quit IRC | 12:15 | |
| *** MaxV has joined #openstack-infra | 12:15 | |
| *** pblaho has joined #openstack-infra | 12:16 | |
| *** pblaho has quit IRC | 12:17 | |
| *** pblaho has joined #openstack-infra | 12:17 | |
| *** pblaho has left #openstack-infra | 12:17 | |
| *** pblaho has joined #openstack-infra | 12:18 | |
| *** pblaho has quit IRC | 12:19 | |
| *** teran has joined #openstack-infra | 12:19 | |
| *** pblaho has joined #openstack-infra | 12:19 | |
| *** teran_ has joined #openstack-infra | 12:20 | |
| openstackgerrit | Denis M. proposed openstack-infra/project-config: Fix Trove gate timeout https://review.openstack.org/135263 | 12:23 |
| *** dmakogon_ is now known as denis_makogon | 12:23 | |
| *** teran has quit IRC | 12:24 | |
| *** gokrokve has joined #openstack-infra | 12:24 | |
| *** gokrokve has quit IRC | 12:29 | |
| *** talluri has joined #openstack-infra | 12:32 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: oslo.messaging: typo fix https://review.openstack.org/135266 | 12:32 |
| *** chandan_kumar has quit IRC | 12:32 | |
| *** pblaho has quit IRC | 12:33 | |
| *** oomichi has quit IRC | 12:33 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 12:33 |
| *** pblaho has joined #openstack-infra | 12:33 | |
| *** pblaho has quit IRC | 12:33 | |
| *** pblaho has joined #openstack-infra | 12:34 | |
| openstackgerrit | Denis M. proposed openstack-infra/project-config: Fix oslo.messaging funtional tests job template https://review.openstack.org/135268 | 12:34 |
| *** doug-fish has joined #openstack-infra | 12:39 | |
| *** talluri_ has joined #openstack-infra | 12:40 | |
| *** talluri has quit IRC | 12:40 | |
| denis_makogon | sdague, ping | 12:41 |
| *** ayoung-dadmode has quit IRC | 12:42 | |
| *** koolhead17 has quit IRC | 12:43 | |
| *** ryanpetrello has joined #openstack-infra | 12:44 | |
| *** koolhead17 has joined #openstack-infra | 12:44 | |
| *** jgallard has joined #openstack-infra | 12:46 | |
| *** mriedem has joined #openstack-infra | 12:46 | |
| *** AlexF has joined #openstack-infra | 12:46 | |
| *** koolhead17 has quit IRC | 12:49 | |
| *** chandan_kumar has joined #openstack-infra | 12:50 | |
| *** chandan_kumar has quit IRC | 12:50 | |
| *** chandankumar has joined #openstack-infra | 12:50 | |
| *** ryanpetrello has quit IRC | 12:51 | |
| *** mbacchi has joined #openstack-infra | 12:51 | |
| *** ryanpetrello has joined #openstack-infra | 12:57 | |
| *** baoli has joined #openstack-infra | 12:57 | |
| *** k4n0 has quit IRC | 13:01 | |
| sdague | denis_makogon: just ask your question | 13:01 |
| denis_makogon | sdague, sure, in your opinion, what should we do with our long-running tests(https://review.openstack.org/#/c/135263/1) ? | 13:01 |
| *** alexpilotti has quit IRC | 13:01 | |
| *** talluri_ has quit IRC | 13:02 | |
| *** radez_g0n3 is now known as radez | 13:02 | |
| sdague | denis_makogon: so it doesn't seem like waiting 3 hrs for test results on patches is really a managable code flow. Breaking up the tests to figure out if there are smaller testable parts is probably worth while | 13:03 |
| *** alexpilotti has joined #openstack-infra | 13:03 | |
| *** koolhead17 has joined #openstack-infra | 13:04 | |
| sdague | without completely diving through trove internal architecture I probably don't have concrete ideas for you. However test runs that exceed an hour really shouldn't be pre-gating | 13:04 |
| denis_makogon | sdague, so, you want to say that we should split our tests into two suits, and run them as two different jobs in parallel, right? | 13:04 |
| sdague | not really | 13:04 |
| sdague | what are the steps in the replication tests? why do they take so long? what optimizations and contracts can be done. | 13:06 |
| *** talluri has joined #openstack-infra | 13:07 | |
| denis_makogon | sdague, replication does automated mysql replication procedure that requires pushing data into Swift, spinning up new instance, applying swift data into it | 13:07 |
| *** nuritv has quit IRC | 13:07 | |
| *** jawed has joined #openstack-infra | 13:07 | |
| denis_makogon | and of course verifying that replication workflow was executed correctly | 13:08 |
| *** kgiusti has joined #openstack-infra | 13:10 | |
| sdague | ok, like I said, I don't have the ability to break that up for you without learning all of trove and proposing more surgical test steps there. However while Trove is part of the integrated gate, delays in it's tests block merging patches in others, so a 3 hr time limit isn't an option. | 13:11 |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add API call to return task statuses https://review.openstack.org/135221 | 13:12 |
| *** mfink has quit IRC | 13:12 | |
| dims | sdague: would you support this feature? https://review.openstack.org/#/c/123464/ ('Save last setting of Toggle CI button') - been stuck for a while | 13:14 |
| *** fandi has quit IRC | 13:15 | |
| sdague | dims: yeh, need to do a bunch of non code things today though, ping me late in the day | 13:16 |
| *** dims has quit IRC | 13:16 | |
| *** rushiagr is now known as rushiagr_away | 13:16 | |
| *** radez is now known as radez_g0n3 | 13:16 | |
| *** dims has joined #openstack-infra | 13:17 | |
| dims | sdague: ack. thanks | 13:17 |
| *** talluri has quit IRC | 13:17 | |
| *** bswartz has quit IRC | 13:22 | |
| *** che-arne has joined #openstack-infra | 13:22 | |
| *** dkranz has joined #openstack-infra | 13:23 | |
| *** gokrokve has joined #openstack-infra | 13:23 | |
| *** pblaho_ has joined #openstack-infra | 13:25 | |
| *** pblaho has quit IRC | 13:26 | |
| *** teran has joined #openstack-infra | 13:27 | |
| *** bhunter71 has joined #openstack-infra | 13:27 | |
| *** teran__ has joined #openstack-infra | 13:30 | |
| *** teran_ has quit IRC | 13:30 | |
| *** mfink has joined #openstack-infra | 13:31 | |
| *** teran has quit IRC | 13:33 | |
| *** weshay has joined #openstack-infra | 13:35 | |
| *** talluri has joined #openstack-infra | 13:35 | |
| *** mpaolino has joined #openstack-infra | 13:35 | |
| *** pblaho__ has joined #openstack-infra | 13:37 | |
| *** pblaho_ has quit IRC | 13:38 | |
| *** stevemar has joined #openstack-infra | 13:38 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack https://review.openstack.org/135215 | 13:39 |
| *** bknudson has joined #openstack-infra | 13:41 | |
| *** talluri has quit IRC | 13:43 | |
| *** nosnos has quit IRC | 13:44 | |
| *** jamespage_ has joined #openstack-infra | 13:46 | |
| *** jamespage_ has quit IRC | 13:46 | |
| *** dkliban_afk is now known as dkliban | 13:48 | |
| *** pblaho has joined #openstack-infra | 13:48 | |
| *** pblaho__ has quit IRC | 13:48 | |
| *** bo_sh has quit IRC | 13:48 | |
| *** hdd has quit IRC | 13:49 | |
| *** pblaho has quit IRC | 13:51 | |
| openstackgerrit | Mike Heald proposed openstack-infra/system-config: Add module for deduplicating files on a path https://review.openstack.org/135287 | 13:51 |
| *** pblaho has joined #openstack-infra | 13:52 | |
| *** krtaylor has quit IRC | 13:55 | |
| *** dkranz has quit IRC | 13:55 | |
| *** bo_sh has joined #openstack-infra | 13:56 | |
| clarkb | afazekas can you review the stack of changes on you multihost d-g change? | 13:57 |
| clarkb | I will approve your change this morning | 13:57 |
| clarkb | but would be good to have your input on the additional updates | 13:57 |
| rcarrill` | hi there | 13:58 |
| *** zul has joined #openstack-infra | 13:58 | |
| rcarrill` | may i please get a core reviewer look at the following changes, it's pending just one +2: | 13:58 |
| *** rcarrill` is now known as rcarrillocruz | 13:58 | |
| *** banix has joined #openstack-infra | 13:58 | |
| rcarrillocruz | https://review.openstack.org/#/c/132194/ | 13:58 |
| rcarrillocruz | https://review.openstack.org/#/c/132283/ | 13:58 |
| *** nadya_ has quit IRC | 13:59 | |
| *** nadya_ has joined #openstack-infra | 14:00 | |
| *** bcourt has quit IRC | 14:01 | |
| *** bswartz has joined #openstack-infra | 14:01 | |
| sdague | dims: you need an infra person for the final +2, I'm only +1 on that repo | 14:01 |
| *** miqui has joined #openstack-infra | 14:01 | |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| *** marun has joined #openstack-infra | 14:02 | |
| fungi | dims: lgtm. i'll need to make sure to touch the header as soon as that merges though | 14:02 |
| *** rushiagr_away is now known as rushiagr | 14:02 | |
| clarkb | afazekas in particular the change to update known hosts files | 14:03 |
| *** krtaylor has joined #openstack-infra | 14:03 | |
| afazekas | clarkb: I added a comment | 14:03 |
| clarkb | awesome thank you /me looks | 14:04 |
| *** mattfarina has joined #openstack-infra | 14:04 | |
| *** nadya_ has quit IRC | 14:04 | |
| afazekas | clarkb: I planned to setup the dev/null as know hosts file, but looks like this part is not in the current code | 14:04 |
| *** bcourt has joined #openstack-infra | 14:05 | |
| *** dprince has joined #openstack-infra | 14:05 | |
| *** ddieterly has joined #openstack-infra | 14:05 | |
| afazekas | Originally that a 3 line double for loop handled the know hosts file with the iptables settings as well. | 14:06 |
| dims | sdague: fungi: thanks! | 14:06 |
| clarkb | afazekas: I think we should do our best to set up known hosts properly | 14:07 |
| clarkb | afazekas: which jogo's change does now | 14:07 |
| clarkb | also I don't think we need that in the base change and I still plan to approve your change as is in a little bit | 14:07 |
| *** jistr has joined #openstack-infra | 14:08 | |
| *** jistr is now known as jistr|mtgs | 14:08 | |
| afazekas | clarkb: how jogo's commit ensures the stack user from the primary node knows about all other hosts ? | 14:09 |
| clarkb | afazekas: it puts the host keys in /etc/ssh/ssh_known_hosts so it is system wide for all users | 14:09 |
| afazekas | clarkb: on the subnodes | 14:09 |
| *** bo_sh has left #openstack-infra | 14:10 | |
| clarkb | afazekas: it does primary node and subnodes | 14:10 |
| clarkb | afazekas: it creates the file on the primary node, then copies that to /etc/ssh/ssh_known_hosts on all of the nodes | 14:10 |
| clarkb | afazekas: https://review.openstack.org/#/c/134670/12/devstack-vm-gate.sh line 468 is where it does it for the primary node | 14:10 |
| *** dustins has joined #openstack-infra | 14:11 | |
| *** yfried is now known as yfried|afk | 14:11 | |
| *** changbl has quit IRC | 14:11 | |
| *** markmcclain has joined #openstack-infra | 14:12 | |
| *** jgrimm is now known as zz_jgrimm | 14:12 | |
| anteaya | I'm about to approve a patch to create a new repo, there were some hiccups with new project creation in the recent past, anything anyone wants me to do before I approve? | 14:13 |
| *** amotoki_ has joined #openstack-infra | 14:13 | |
| *** cdent has joined #openstack-infra | 14:15 | |
| *** ildikov has quit IRC | 14:15 | |
| anteaya | stevebaker: any thoughts about making this job experimental or non-voting whilst you working out any kinks with dib enabled? https://review.openstack.org/#/c/134422/1 | 14:16 |
| *** cpowell has joined #openstack-infra | 14:17 | |
| *** andreykurilin_ has joined #openstack-infra | 14:17 | |
| *** andreykurilin_ has quit IRC | 14:22 | |
| *** andreykurilin_ has joined #openstack-infra | 14:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Make existing requirements parsing more forgiving https://review.openstack.org/134637 | 14:24 |
| mmedvede | anteaya, yesterday the new irc-meetings project has taken about 2.5 hours to be properly created. I have brought it up with fungi initially. Not sure if he had to do anything manually | 14:24 |
| *** dmsimard_away is now known as dmsimard | 14:24 | |
| fungi | mmedvede: yeah, i couldn't see any obvious errors from when it should have initially been created. retriggering replication seemed to get the commits to appear in our git farm though | 14:25 |
| fungi | would be good to keep an eye out for similar incidents so we can correlate and possibly narrow down the cause | 14:25 |
| *** ayoung has joined #openstack-infra | 14:27 | |
| anteaya | fungi: did you want to watch this one move through? | 14:27 |
| anteaya | mmedvede: yes that was the project I was thinking of, thanks for reminding me of the name | 14:28 |
| clarkb | I just approved afazekas' d-g change | 14:28 |
| fungi | anteaya: not sure there's much to watch, but if it doesn't show up with any commits on git.openstack.org then i'll see if i can find more details | 14:28 |
| clarkb | once that is in I will follow up with a layout.yaml change to make that a check test against d-g (does anyone want it to be gating as well?) | 14:29 |
| anteaya | okay, I'll go ahead and approve it | 14:29 |
| *** mattfarina has quit IRC | 14:30 | |
| *** ildikov has joined #openstack-infra | 14:31 | |
| *** yfried|afk is now known as yfried | 14:31 | |
| *** andreykurilin__ has joined #openstack-infra | 14:33 | |
| *** andreykurilin_ has quit IRC | 14:33 | |
| *** dkranz has joined #openstack-infra | 14:34 | |
| *** koolhead17 has quit IRC | 14:35 | |
| *** che-arne has quit IRC | 14:36 | |
| mordred | clarkb: I just realized - why are you awake? | 14:37 |
| clarkb | mordred: the jetlag is strong | 14:37 |
| mordred | clarkb: ah. condolensces | 14:37 |
| mordred | this is clearly not a time that a clarkb norally enjoys being awake | 14:37 |
| clarkb | I fell asleep at about 7pm last night. woke up at 3:30am after ~8 hours of sleep. Had breakfast then IRC | 14:37 |
| clarkb | mordred: the worst part is nothing is open this early in the morning. I could got to sheris and thats about it | 14:38 |
| *** ildikov has quit IRC | 14:38 | |
| mordred | clarkb: that's my least favorite part too | 14:38 |
| mordred | clarkb: I mean, even in New York, when you wake up at 3:30am, the things that are open are not really the things you want to go to in a jetlagged state | 14:38 |
| clarkb | I had to wait about 8 hours yesterday before I could get a burrito | 14:39 |
| *** dmellado has quit IRC | 14:39 | |
| mordred | that's the worst | 14:39 |
| clarkb | post europe burrito was amazing. Got it smothered in mole poblano. Finally some food with a little heat | 14:39 |
| *** dmellado has joined #openstack-infra | 14:40 | |
| clarkb | it was so good I am really tempted to go back for another today | 14:40 |
| anteaya | clarkb: noone in paris understood heat? | 14:40 |
| *** che-arne has joined #openstack-infra | 14:41 | |
| clarkb | anteaya: european food is good, but is built on salt and fat. I don't think they know what peppers are | 14:41 |
| fungi | parisian cuisine was remarkably lacking in foods spicy enough make your mouth numb | 14:41 |
| *** yfried is now known as yfried|afk | 14:41 | |
| anteaya | clarkb: yes I do recall the salt and fat, saw red peppers but they were sweet | 14:41 |
| *** hashar has joined #openstack-infra | 14:42 | |
| anteaya | fungi: true I have no memories of numb mouth | 14:42 |
| clarkb | so there was a craving for new world food | 14:42 |
| anteaya | I can hear that, yes | 14:43 |
| *** david-lyle_afk is now known as david-lyle | 14:43 | |
| *** jheroux has joined #openstack-infra | 14:43 | |
| clarkb | if I go back and get a chimichanga today instead of a burrito that may justify going back today | 14:43 |
| fungi | had a big bowl of habanero chili first thing when i got back. it was goood | 14:43 |
| anteaya | clarkb: what is the latest state of agreement about goes in the gate test pipeline and what goes in the check test pipeline? | 14:43 |
| *** amitgandhinz has joined #openstack-infra | 14:43 | |
| clarkb | anteaya: typically we try to keep them inline for voting tests | 14:43 |
| anteaya | clarkb: you have to justify your food choices? | 14:43 |
| anteaya | clarkb: okay thanks | 14:44 |
| clarkb | anteaya: only because I am tired :) | 14:44 |
| anteaya | clarkb: ah justify away then | 14:44 |
| anteaya | if I have a craving I just listen to it | 14:44 |
| anteaya | I figure there must be a reason and I'll figure out what it is after I am full | 14:44 |
| clarkb | fungi: that sounds good | 14:46 |
| *** chandankumar has quit IRC | 14:46 | |
| *** emagana has joined #openstack-infra | 14:46 | |
| *** yfried|afk is now known as yfried | 14:47 | |
| *** ildikov has joined #openstack-infra | 14:51 | |
| *** jamespage_ has joined #openstack-infra | 14:52 | |
| *** jamespage_ has quit IRC | 14:52 | |
| *** rakhmerov has joined #openstack-infra | 14:53 | |
| openstackgerrit | Merged openstack-infra/system-config: Save last setting of Toggle CI button https://review.openstack.org/123464 | 14:53 |
| *** funzo has joined #openstack-infra | 14:54 | |
| clarkb | fungi: ^ I assume you will be restarting gerrit for that? | 14:54 |
| clarkb | or did we fix the issues around that? | 14:54 |
| fungi | i'll touch GerritSiteHeader.html on review.o.o as soon as that gets puppeted in | 14:54 |
| fungi | that's supposed to solve the issue | 14:54 |
| clarkb | fungi: do we need to update our puppet manifests to do that? | 14:54 |
| fungi | clarkb: perhaps | 14:55 |
| clarkb | I am still not sure I understand how that all broke ast time | 14:55 |
| clarkb | (which is why I have not been reviewing js changes for gerrit) | 14:55 |
| fungi | last time i got paranoid because i was on a crappy ancient netbook with too old of a browser to support the hideci stuff, but it wasn't actually broken. the time before that i was confused as to what file i was supposed to touch | 14:56 |
| openstackgerrit | Merged openstack-infra/project-config: Restart ssh after configuring sshd_config https://review.openstack.org/134371 | 14:56 |
| *** katyafervent has joined #openstack-infra | 14:56 | |
| clarkb | fungi: puppet touches that file for us | 14:57 |
| clarkb | when it actually broke that time you touched manually? | 14:57 |
| *** dteselkin has joined #openstack-infra | 14:58 | |
| *** ZZelle has quit IRC | 14:58 | |
| fungi | yeah. could be a race there i suppose | 14:58 |
| *** teran__ has quit IRC | 14:58 | |
| *** ZZelle has joined #openstack-infra | 14:58 | |
| clarkb | I don't think it was racing, I think we touched the wrong fiel | 14:58 |
| clarkb | but I am not sure if we touched the wrong file for the data being updated or if it was a manual touch or if puppet was just wrong | 14:59 |
| fungi | well, i suppose we'll be able to tell this time around | 14:59 |
| clarkb | ya | 14:59 |
| *** tsufiev has joined #openstack-infra | 14:59 | |
| fungi | any moment now | 14:59 |
| *** teran has joined #openstack-infra | 14:59 | |
| *** bhuvan has joined #openstack-infra | 14:59 | |
| fungi | mtreinish: i've set up a puppet dev env to deploy the subunit worker out of, so we can end the death by a thousand patches | 15:00 |
| *** teran_ has joined #openstack-infra | 15:00 | |
| fungi | mtreinish: i'll let you know in a sec if your latest patch is sufficient, or extend it for you if not | 15:00 |
| mtreinish | fungi: heh ok awesome (although I was enjoying the stat padding :) | 15:00 |
| mtreinish | fungi: ok cool, let me know | 15:01 |
| fungi | huh... it finished, no errors | 15:01 |
| fungi | i'll get it into dns in a moment | 15:02 |
| *** nadya_ has joined #openstack-infra | 15:03 | |
| *** teran has quit IRC | 15:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Move puppet-monasca to Stackforge https://review.openstack.org/134658 | 15:04 |
| *** openstackgerrit has quit IRC | 15:04 | |
| *** openstackgerrit has joined #openstack-infra | 15:04 | |
| *** armax has joined #openstack-infra | 15:04 | |
| fungi | okay, hideci.js has a new timestamp and is no longer working in the gerrit ui. now to try bringing it back | 15:05 |
| *** SumitNaiksatam has quit IRC | 15:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add barbican-cover to check pipeline https://review.openstack.org/133582 | 15:05 |
| *** zz_dimtruck is now known as dimtruck | 15:05 | |
| *** marun has quit IRC | 15:06 | |
| fungi | though worth noting, puppet _does_ seem to have updated the timestamp on GerritSiteHeader.html at roughly the same time | 15:06 |
| clarkb | I think we need to touch a different file then? | 15:07 |
| fungi | i have proceeded to touch it as well, and so far no effect | 15:07 |
| fungi | no, wait, it's back now | 15:07 |
| clarkb | maybe the css file? | 15:07 |
| clarkb | oh | 15:07 |
| fungi | so there may be a need to introduce a delay | 15:07 |
| *** jawed has quit IRC | 15:07 | |
| *** Sukhdev has joined #openstack-infra | 15:07 | |
| clarkb | a sleep 1 in the puppet touch exec? | 15:08 |
| fungi | not sure whether it took several minutes for puppet touching that file to take effect, or me touching it a second time a couple minutes later | 15:08 |
| *** radez_g0n3 is now known as radez | 15:08 | |
| fungi | though there was also a noticeable delay, say 30 seconds (perhaps a caching layer in front had to refresh), once i touched the file | 15:09 |
| *** MaxV has quit IRC | 15:09 | |
| *** _buzz has quit IRC | 15:09 | |
| fungi | dims: the cookie does seem to be effective across tabs, as intended | 15:10 |
| *** _buzz has joined #openstack-infra | 15:11 | |
| *** ChuckC_ has joined #openstack-infra | 15:11 | |
| clarkb | ya it seems to be working for me | 15:11 |
| *** ChuckC has quit IRC | 15:11 | |
| mtreinish | fungi: cool, so everything worked, now I guess it's more a test if the worker daemons do the right thing... | 15:11 |
| clarkb | fungi: krotscheck: ok I have multihost d-g mostly dealt with. On to storyboard. Should I be approving the docs updates as I go? | 15:12 |
| fungi | mtreinish: yep. once that one-liner patch of yours merges, i'll enable puppet updates for that worker | 15:12 |
| clarkb | and should we draft and announcement email? | 15:12 |
| clarkb | and fungi said he would do lp at some point (happy to help with that too) | 15:13 |
| fungi | clarkb: i think krotscheck said he was drafting the announcement | 15:13 |
| *** banix has quit IRC | 15:13 | |
| clarkb | perfect | 15:13 |
| fungi | and yeah, i need to do final passes with the import script before closing down bug reporting on any projects | 15:13 |
| hashar | fungi: clarkb: the GerritSiteHeader.html and other css are rendered inline by Gerrit which keep its own cache version of them | 15:13 |
| hashar | iirc its cached for 5 minutes | 15:13 |
| fungi | hashar: aha, so perhaps i just didn't wait long enough | 15:14 |
| *** jedimike has quit IRC | 15:14 | |
| clarkb | hashar: that would explain it | 15:14 |
| *** AlexF has quit IRC | 15:14 | |
| fungi | clarkb: i guess next time let's give it something like 10 minutes from the js file update before giving up and souching the site header file | 15:14 |
| fungi | s/souching/touching/ | 15:15 |
| *** MaxV has joined #openstack-infra | 15:15 | |
| dims | fungi: i opened 1 tab, make sure i don't have any jenkins runs, then i opened 2 more, (no jenkins runs on all 3), then i click toggle button on one of them and refresh the other 2. should see jenkins runs on all 3 | 15:15 |
| clarkb | fungi: sounds like a plan | 15:16 |
| fungi | dims: yep, that's what i tried | 15:16 |
| fungi | dims: seems it works | 15:16 |
| *** bknudson has quit IRC | 15:16 | |
| *** nikhil_k|afk is now known as nikhil_k | 15:16 | |
| dims | fungi: cool | 15:16 |
| hashar | clarkb: fungi: I am wrong, should be more or less immediate, well I guess whenever Gerrit figure out the files have been changed on disk and regenerate the cache file and inlined versions . files under /static/ have a 5 minutes expire. https://review.openstack.org/Documentation/config-themes.html#_http_caching | 15:16 |
| fungi | hashar: ahh, right, this was a file under static | 15:17 |
| fungi | hashar: in the past it's appeared that gerrit doesn't notice changes to files in static unless a file in etc gets updated triggering it to rebuild its bundled content | 15:18 |
| *** eharney has joined #openstack-infra | 15:18 | |
| *** _shaps_ has joined #openstack-infra | 15:19 | |
| *** andymaier has joined #openstack-infra | 15:20 | |
| *** MaxV has quit IRC | 15:21 | |
| *** ChuckC_ has quit IRC | 15:21 | |
| *** MaxV has joined #openstack-infra | 15:21 | |
| *** AlexF has joined #openstack-infra | 15:22 | |
| mrmartin | clarkb, fungi: hi | 15:22 |
| clarkb | hello | 15:22 |
| mrmartin | I like to get to ssl certs for groups.openstack.org and groups-dev.openstack.org. https://blueprints.launchpad.net/openstack-ci/+spec/groups-oauth2-authentication :) | 15:23 |
| *** sweston_ is now known as sweston | 15:23 | |
| mrmartin | I've switched the auth mechanism from openid to oauth2, because openid.sreg was missing some key fields that required for retrieve picture urls. | 15:24 |
| mrmartin | but openstackid is enforcing ssl for two-way communication between the oauth2 provider and third party app. | 15:24 |
| *** xyang0 has joined #openstack-infra | 15:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Add missing logstash import https://review.openstack.org/135145 | 15:25 |
| *** nelsnelson has joined #openstack-infra | 15:26 | |
| fungi | mrmartin: oh, so we do have oauth2 support in openstackid? i thought that got ripped out when they shifted gears to implement openid there | 15:26 |
| *** amuller__ has joined #openstack-infra | 15:26 | |
| mrmartin | we have. :) | 15:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Flush out the db api documenation https://review.openstack.org/129405 | 15:26 |
| *** sabeen has quit IRC | 15:27 | |
| mrmartin | I made a dev-vagrant box for openstackid, that works in my dev environment, and I wrote an oauth2 module for drupal, it seems to be working well. | 15:28 |
| fungi | mrmartin: i can go ahead and buy one for groups.openstack.org, but can't we configure openstackid-dev.openstack.org to trust the self-signed cert groups-dev.openstack.org is using? | 15:28 |
| *** amuller_ has quit IRC | 15:28 | |
| *** zz_jgrimm is now known as jgrimm | 15:28 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Reorganize the python api documentation https://review.openstack.org/129406 | 15:28 |
| mrmartin | fungi, I gues self-signed cert is totally enough for groups-dev, the only thing we need to check, maybe we need to inject to self-signed cert as a root-one into openstackid-dev | 15:28 |
| fungi | mrmartin: right, probably need to add it to wherever openstackid is looking for its trust chain | 15:29 |
| mrmartin | don't forget that. | 15:29 |
| fungi | /etc/ssl/certs or similar | 15:29 |
| *** marun has joined #openstack-infra | 15:29 | |
| mrmartin | yep, I'm working on the required system-config patch, if everything goes well, I'll publish that tomorrow. | 15:29 |
| sweston | ttx: ping, hello, when you have a moment, could we get your input on this? https://review.openstack.org/#/c/135170/ the question has been raised on the proper location for this spec ... is this the correct location for it, or does it qualify for cross project? | 15:29 |
| fungi | mrmartin: sounds good. i'll add the ssl cert to my to do list | 15:29 |
| *** cdent has quit IRC | 15:30 | |
| mrmartin | ok, thanks, let's focus on the groups-dev one, this is required for testing at staging server, and if it works well, we can roll-out to prod server with a real certificate. | 15:30 |
| anteaya | sweston: did you want to clean up the whitespace errors on that patch? and perhaps fill in some values for the license header? | 15:31 |
| *** bhuvan has quit IRC | 15:32 | |
| *** bdpayne has joined #openstack-infra | 15:32 | |
| clarkb | does openid not need ssl certs to be trusted? | 15:32 |
| clarkb | and if so isn't that a bug? | 15:32 |
| sweston | anteaya: arg, yes, I didn't notice that until this morning .. was pretty tired when I submitted this | 15:33 |
| *** jawed has joined #openstack-infra | 15:33 | |
| fungi | clarkb: i think openid doesn't require ssl on the consumer, just on the provider | 15:33 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add section on the basic data model https://review.openstack.org/129407 | 15:33 |
| fungi | clarkb: since the consumer redirects to the provider and consumes a redirect from the provider post-authentication | 15:33 |
| clarkb | fungi: but if ssl is there shouldn't it require things be trusted? | 15:33 |
| anteaya | sweston: figured as much, which is why I didn't -1, you just needed to take another look | 15:33 |
| clarkb | maybe not | 15:33 |
| clarkb | its early in the morning and I am jetlagged :P | 15:34 |
| *** Sukhdev has quit IRC | 15:34 | |
| fungi | but all authentication happens through interaction with the provider, which is using a globally-trusted ssl cert | 15:34 |
| *** jamespage_ has joined #openstack-infra | 15:34 | |
| clarkb | gotcha | 15:34 |
| sweston | anteaya: thank you :-D | 15:35 |
| *** bknudson has joined #openstack-infra | 15:35 | |
| *** andreykurilin__ has quit IRC | 15:36 | |
| *** jerryz has joined #openstack-infra | 15:36 | |
| *** andreykurilin_ has joined #openstack-infra | 15:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Make bashate voting on system-config https://review.openstack.org/129667 | 15:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Fix subunit-gearman-worker.py path https://review.openstack.org/135329 | 15:37 |
| mtreinish | fungi: ^^^ found another bug by looking at puppetboard... | 15:37 |
| mtreinish | not sure why that didn't cause an explosion | 15:38 |
| mrmartin | clarkb: oauth2 requires that, it have a two-way communication, and sends back an authorization code to the third-party webapp | 15:39 |
| *** wenlock has joined #openstack-infra | 15:40 | |
| anteaya | sweston: :D | 15:43 |
| *** bdpayne has quit IRC | 15:44 | |
| *** otherwiseguy has joined #openstack-infra | 15:45 | |
| sweston | anteaya: I do have a question about the copyright, though ... not quite sure what to do with that section | 15:46 |
| anteaya | sweston: good question, but I have no insight, perhaps someone else does | 15:46 |
| *** zz_gondoi is now known as gondoi | 15:47 | |
| *** koolhead17 has joined #openstack-infra | 15:47 | |
| *** vigneshvar has quit IRC | 15:48 | |
| *** afazekas has quit IRC | 15:49 | |
| openstackgerrit | Jukka Lehtniemi proposed openstack-infra/zuul: Add support for Gerrit replication events https://review.openstack.org/135332 | 15:49 |
| sweston | anteaya: crickets ... perhaps somebody will answer it in a bit, let me google around some more | 15:51 |
| anteaya | sweston: go forth bravely, I think everyone is otherwise occupied | 15:51 |
| *** markmcclain has quit IRC | 15:52 | |
| sweston | anteaya: hehe, bravely ... | 15:53 |
| ttx | sweston: hmm | 15:54 |
| *** chandankumar has joined #openstack-infra | 15:54 | |
| ttx | sweston: this feels like an infrastructure spec to me | 15:54 |
| ttx | sweston: is there anything that will need to land in code in other projects ? | 15:54 |
| *** AlexF has quit IRC | 15:54 | |
| *** ociuhandu has joined #openstack-infra | 15:54 | |
| *** mrmartin has quit IRC | 15:54 | |
| sweston | ttx: ok. we just wanted to consult with you to make sure it is in the right place | 15:54 |
| ttx | cross-project specs are designs where implementations will likely happen everywhere | 15:55 |
| *** erikwilson has joined #openstack-infra | 15:55 | |
| ttx | here it feels more like a dashboard offered to every project | 15:55 |
| *** nadya_ has quit IRC | 15:55 | |
| ttx | it doesn't really require approval from all affected PTls either | 15:56 |
| *** yfried has quit IRC | 15:57 | |
| sweston | ttx: there is no cross project code requirements right now, though in the future the direction might be different .. the idea of ci "calibration" has been explored, but it introduces other complexities | 15:57 |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add timeout to the blur event of tag-complete https://review.openstack.org/135334 | 15:57 |
| ttx | sweston: ack | 15:58 |
| sweston | ttx: ok, thanks for your input on this. | 15:58 |
| *** SumitNaiksatam has joined #openstack-infra | 15:58 | |
| ttx | sweston: np! | 15:58 |
| *** sabeen1 has joined #openstack-infra | 15:59 | |
| *** baoli has quit IRC | 16:00 | |
| anteaya | mtreinish and jogo would really like your input on this patch about filtering tests for nova cells: https://review.openstack.org/#/c/135064/1 | 16:00 |
| mtreinish | anteaya: I thought I already -1'd a patch like that | 16:04 |
| mtreinish | from johnthetubaguy IIRC | 16:04 |
| *** shashankhegde has joined #openstack-infra | 16:04 | |
| johnthetubaguy | I did create one, it got lots of negative, so I abandoned it | 16:05 |
| *** yamahata has quit IRC | 16:05 | |
| mtreinish | anteaya: https://review.openstack.org/#/c/125980/ | 16:05 |
| *** KanagarajM has joined #openstack-infra | 16:05 | |
| anteaya | mtreinish johnthetubaguy thank you for letting me know | 16:06 |
| anteaya | I will comment as such so now andrew will know as well | 16:06 |
| johnthetubaguy | right now nova-cells is not allowing any VMs to boot, but the flavor work thats happening should fix that | 16:06 |
| *** pcrews has joined #openstack-infra | 16:06 | |
| *** mpaolino has quit IRC | 16:07 | |
| *** baoli has joined #openstack-infra | 16:07 | |
| *** gilliard has quit IRC | 16:07 | |
| mtreinish | johnthetubaguy: btw, I never did respond to your comment there, I wasn't saying we should not skip those tests, but that you should do it through the config file in devstack | 16:07 |
| mtreinish | because using a regex is error prone and can cause issues (and has) when new tests are added | 16:08 |
| *** fandi has joined #openstack-infra | 16:08 | |
| johnthetubaguy | mtreinish: I agreed with your comment, talking to sdague we might do the regex for now, because its more a keep it going for a bit, while we fix code and evolve the set of tests | 16:09 |
| johnthetubaguy | the summit and other fires have meant I haven't got back to the cells tests, alaski is leading up the cells sub group that should be driving this going forward | 16:10 |
| *** aysyd has quit IRC | 16:10 | |
| *** denis_makogon has quit IRC | 16:10 | |
| sdague | mtreinish: so, realistically, this is stop gap | 16:10 |
| anteaya | I'm fine with whatever the direction is, let's just have some consenseus around it | 16:10 |
| *** mpaolino has joined #openstack-infra | 16:11 | |
| sdague | regex is going to be there just long enough to not let this slide into the sea before we cut over to cells v2 | 16:11 |
| sdague | then we shoot it in the head | 16:11 |
| sdague | given the cells v2 plan, I don't think doing the cells testing correctly actually has a lot of benefit | 16:11 |
| *** KanagarajM has quit IRC | 16:12 | |
| johnthetubaguy | yeah, it more stop whats there rotting too bad if we can, but yeah, its not a long term thing | 16:12 |
| alaski | right. I put that up because of a discussion at the summit. If we were planning to keep cells around I would certainly tackle it differently | 16:12 |
| alaski | keep *current* cells around | 16:13 |
| *** marun has quit IRC | 16:13 | |
| fungi | it does appear that stackforge/puppet-monasca was created on git.o.o as an uninitialized repo, so taking a look now to see if i can figure out why | 16:13 |
| mtreinish | ok, sure that's fine I guess, I was just concerned that this was another attempt to short circuit things | 16:13 |
| anteaya | fungi: thanks | 16:13 |
| anteaya | alaski: hi, don't take the -2 personally, in future if you are respinning an abandoned patch, please link to it if you know about it | 16:14 |
| *** bdpayne has joined #openstack-infra | 16:14 | |
| sdague | mtreinish: nope, alaski, johnthetubaguy, and I discussed during the nova sessions, I think this is a pragmatic plan to keep the eye on the actual ball, which is a cells v2 that supports all nova function | 16:14 |
| sdague | and doesn't completely burn existing users | 16:14 |
| *** MarkAtwood has joined #openstack-infra | 16:14 | |
| *** esmute has quit IRC | 16:14 | |
| alaski | anteaya: no worries about a -2. but I was not aware of an older patch | 16:14 |
| anteaya | sdague: do you want me to lift my -2? | 16:15 |
| anteaya | alaski: ah okay, thanks | 16:15 |
| mtreinish | anteaya: yeah I think using the regex is fine given this discussion | 16:15 |
| sdague | anteaya: yes please | 16:16 |
| sdague | I have a -1 on a name change I'd like to see there, otherwise I'm +2 on the approach | 16:16 |
| anteaya | sdague: I will lift | 16:17 |
| *** todd_dsm has joined #openstack-infra | 16:17 | |
| *** esmute has joined #openstack-infra | 16:17 | |
| *** kgiusti has left #openstack-infra | 16:17 | |
| fungi | Nov 18 15:12:29 review puppet-agent[32445]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]/returns) 2014-11-18 15:10:37,115: manage_projects - ERROR - Exception processing ACLS for stackforge/puppet-monasca. | 16:18 |
| alaski | sdague: there's a job already defined without the -full name, but it's not used anywhere. I can move this there and then update the projects to use that job | 16:18 |
| fungi | trying again manually to see if it's perma-broken or if that was transient | 16:18 |
| *** belmoreira has quit IRC | 16:18 | |
| * fungi grumbles | 16:19 | |
| fungi | rerunning it by hand, no errors, and worked fine | 16:19 |
| sdague | alaski: that works for me | 16:19 |
| anteaya | fungi: exception processing acls | 16:19 |
| sdague | just want to be descriptive about it | 16:19 |
| alaski | sdague: definitely | 16:19 |
| anteaya | fungi: did we get any more details about the exception? | 16:19 |
| fungi | anteaya: yeah, correlating from gerrit logs now to see if it kept a record of the rejection | 16:20 |
| anteaya | kk | 16:20 |
| *** groknix has joined #openstack-infra | 16:20 | |
| *** dannywilson has joined #openstack-infra | 16:22 | |
| fungi | [2014-11-18 15:10:29,126] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Error creating local repository /opt/lib/git/stackforge/puppet-monasca.git: java.io.IOException: Creating directories for /opt/lib/git/stackforge/puppet-monasca.git failed | 16:22 |
| anteaya | :/ | 16:22 |
| *** dannywilson has quit IRC | 16:22 | |
| *** dannywil_ has joined #openstack-infra | 16:23 | |
| fungi | no idea if that's the cause or just another symptom | 16:23 |
| anteaya | time stamps are close | 16:23 |
| clarkb | fungi: thats on an ssd volume iirc | 16:23 |
| anteaya | the directory failure occured first | 16:23 |
| clarkb | maybe it had a transient write error? | 16:23 |
| clarkb | I would check syslog | 16:23 |
| *** amuller__ has quit IRC | 16:24 | |
| anteaya | would be hard to create acls for a dir that didn't exist | 16:24 |
| *** dizquierdo has quit IRC | 16:24 | |
| fungi | nah, that's the local replica not the repo gerrit uses directly | 16:24 |
| anteaya | oh | 16:25 |
| *** aysyd has joined #openstack-infra | 16:26 | |
| *** SumitNaiksatam has quit IRC | 16:26 | |
| *** nadya__ has joined #openstack-infra | 16:26 | |
| fungi | not finding anything in syslog other than that (unfortunately very vague) puppet error | 16:26 |
| *** mpaolino has quit IRC | 16:26 | |
| fungi | well, and: | 16:27 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for conf.sample oslo.db bug 1393559 https://review.openstack.org/135343 | 16:27 |
| fungi | Nov 18 15:12:29 review puppet-agent[32445]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]/returns) CreateGroupException | 16:27 |
| uvirtbot | Launchpad bug 1393559 in heat "oslo.db release breaks sample config" [Undecided,Fix committed] https://launchpad.net/bugs/1393559 | 16:27 |
| fungi | i'll clean up the traceback for that | 16:27 |
| *** banix has joined #openstack-infra | 16:27 | |
| anteaya | fungi: k, also when you have a moment can you disable 10192 cisco_neutron_ci Cisco CI cisco-openstack-neutron-ci@cisco.com | 16:28 |
| anteaya | please? | 16:28 |
| openstackgerrit | Merged openstack-infra/project-config: Add testr logs publishing for manilaclient functional tests job https://review.openstack.org/134938 | 16:28 |
| fungi | anteaya: done | 16:28 |
| anteaya | thanks | 16:28 |
| *** marun has joined #openstack-infra | 16:29 | |
| fungi | so basically this extremely short traceback from manage-projects... http://paste.openstack.org/show/134424/ | 16:30 |
| *** ociuhandu has quit IRC | 16:30 | |
| clarkb | :/ | 16:31 |
| fungi | i guess we could use a little more debugging output from there | 16:32 |
| fungi | oh, wait, false positive | 16:32 |
| fungi | that was one of the various openstack-attic projects. worth noting manage-projects doesn't cope well with read-only projects | 16:33 |
| *** jawed has quit IRC | 16:33 | |
| *** hdd has joined #openstack-infra | 16:33 | |
| anteaya | ah | 16:33 |
| anteaya | are the openstack-attic projects tripping up the rest of manage-projects operations? | 16:33 |
| *** gokrokve_ has joined #openstack-infra | 16:34 | |
| *** sergsh is now known as sergsh_away | 16:34 | |
| *** nadya__ has quit IRC | 16:34 | |
| fungi | nah, just adding a lot of noise | 16:34 |
| anteaya | kk | 16:35 |
| *** KanagarajM has joined #openstack-infra | 16:35 | |
| *** ociuhandu has joined #openstack-infra | 16:35 | |
| *** unicell has joined #openstack-infra | 16:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add timeout to the blur event of tag-complete https://review.openstack.org/135334 | 16:36 |
| *** aysyd has quit IRC | 16:37 | |
| *** jamespage_ has quit IRC | 16:37 | |
| *** gokrokve has quit IRC | 16:37 | |
| fungi | aha, here we go! | 16:37 |
| fungi | http://paste.openstack.org/show/134438/ | 16:37 |
| anteaya | that MySQL server | 16:38 |
| anteaya | it needs to stick around | 16:38 |
| fungi | that's what's thrown when it tried to add the new project | 16:38 |
| fungi | checking timestamps now | 16:38 |
| anteaya | k | 16:38 |
| *** thedodd has joined #openstack-infra | 16:38 | |
| *** mpaolino has joined #openstack-infra | 16:38 | |
| *** ildikov has quit IRC | 16:38 | |
| *** gokrokve_ has quit IRC | 16:38 | |
| *** lttrl has quit IRC | 16:38 | |
| *** yamamoto has quit IRC | 16:39 | |
| *** zul has quit IRC | 16:39 | |
| *** changbl has joined #openstack-infra | 16:39 | |
| *** lttrl has joined #openstack-infra | 16:39 | |
| fungi | ansible-puppet reports starting at 15:04:09 and that exception is raised at 15:12:29 | 16:40 |
| anteaya | hmmm | 16:40 |
| fungi | so we're talking about less than 10 minutes... does mysql maybe close out idle sockets very aggressively? | 16:41 |
| anteaya | this I don't know | 16:41 |
| clarkb | oh maybe | 16:41 |
| clarkb | and that may explain paste's occasional issues too | 16:41 |
| anteaya | oh we might be able to fix paste? | 16:41 |
| * anteaya does a happy dance | 16:41 | |
| clarkb | anteaya: well its smeting to do with mysql connectivity | 16:41 |
| clarkb | but its been a low priority to debug | 16:42 |
| anteaya | sounds like it | 16:42 |
| anteaya | we might have fallen over something useful though | 16:42 |
| *** hdd has quit IRC | 16:42 | |
| fungi | so current wild theory, this breaks when the manage-projects run takes too long to complete and loses its db socket | 16:43 |
| fungi | which would explain why it's not every time | 16:43 |
| anteaya | so far the facts support this | 16:43 |
| *** ChuckC_ has joined #openstack-infra | 16:44 | |
| *** ChuckC_ is now known as ChuckC | 16:44 | |
| *** ZZelle has quit IRC | 16:45 | |
| *** aysyd has joined #openstack-infra | 16:48 | |
| *** arxcruz has quit IRC | 16:48 | |
| *** atiwari has joined #openstack-infra | 16:49 | |
| *** Guest67947 has quit IRC | 16:50 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:50 | |
| anteaya | I know zip about MySQL sockets, what can be done? | 16:52 |
| *** ociuhandu has quit IRC | 16:53 | |
| *** liam__ has joined #openstack-infra | 16:53 | |
| clarkb | for manage-projects we can probably connect to the local socket rather than using tcp | 16:53 |
| clarkb | oh wait | 16:53 |
| *** liam__ has quit IRC | 16:53 | |
| clarkb | we use trove, ignore me :) | 16:53 |
| *** aysyd has quit IRC | 16:54 | |
| *** pc_m has quit IRC | 16:54 | |
| anteaya | is this a trove issue masqarading as a MySQL issue? | 16:54 |
| fungi | we might be able to catch that exception, reconnect and retry the operation? | 16:54 |
| *** pc_m has joined #openstack-infra | 16:54 | |
| *** Longgeek has quit IRC | 16:55 | |
| *** ociuhandu has joined #openstack-infra | 16:55 | |
| *** dangers_away is now known as dangers | 16:56 | |
| *** sarob has joined #openstack-infra | 16:56 | |
| clarkb | anteaya: trove runs a real mysql. its likely more a settings thing | 16:58 |
| openstackgerrit | Merged openstack-infra/system-config: Fix subunit-gearman-worker.py path https://review.openstack.org/135329 | 16:58 |
| clarkb | fungi: ya either that or tune mysql to allow us to do these things (catching exception is probably better) | 16:58 |
| *** emagana has quit IRC | 17:00 | |
| *** emagana has joined #openstack-infra | 17:00 | |
| *** hashar has quit IRC | 17:01 | |
| *** jamespage_ has joined #openstack-infra | 17:02 | |
| *** bdpayne has quit IRC | 17:03 | |
| *** andymaier has quit IRC | 17:03 | |
| *** jamespage_ has quit IRC | 17:03 | |
| *** zul has joined #openstack-infra | 17:03 | |
| nibalizer | infra meeting is at 17:00 utc? | 17:03 |
| *** r-daneel has joined #openstack-infra | 17:03 | |
| clarkb | its at 11am pacific, which is 1900 UTC if I have done my math correctly | 17:03 |
| nibalizer | approx 2 hours from now? | 17:03 |
| clarkb | ya | 17:03 |
| nibalizer | i've hit that poitn where too much travel and now everything is confused | 17:04 |
| nibalizer | okay sweet | 17:04 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Filter out some Tempest tests for Nova cells https://review.openstack.org/135064 | 17:04 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Remove -full designation from cells job https://review.openstack.org/135360 | 17:04 |
| clarkb | nibalizer: we miss you to | 17:04 |
| clarkb | nibalizer: there will be mad burrito eating when you get back | 17:04 |
| nibalizer | omg burrito | 17:04 |
| *** r-daneel has quit IRC | 17:04 | |
| *** IvanBerezovskiy has left #openstack-infra | 17:04 | |
| mtreinish | clarkb: what about dst? that switched over during Paris... | 17:04 |
| *** thedodd has quit IRC | 17:05 | |
| clarkb | mtreinish: ya I accounted for that | 17:05 |
| *** emagana has quit IRC | 17:05 | |
| clarkb | it was noon pacific before DST silliness | 17:05 |
| fungi | nibalizer: yes, 19:00 utc. it's been 19:00 utc for several years | 17:05 |
| *** r-daneel has joined #openstack-infra | 17:05 | |
| mtreinish | clarkb: ah, ok. I screwed that all up in my last meeting annoucment email | 17:05 |
| anteaya | nibalizer: hi there | 17:06 |
| anteaya | nibalizer: so puppet-blacksmith gem: https://review.openstack.org/#/c/134834/ | 17:06 |
| fungi | i think it's safer to just announce/document utc meeting times and leave people to their own devices to figure out how to convert between utc and their local times | 17:06 |
| anteaya | utc ftw | 17:06 |
| *** MaxV has quit IRC | 17:06 | |
| *** jcoufal has quit IRC | 17:07 | |
| *** r-daneel has quit IRC | 17:07 | |
| mtreinish | fungi: that's what I used to do and people complained, so I gave in | 17:07 |
| anteaya | mtreinish: softie | 17:07 |
| *** r-daneel has joined #openstack-infra | 17:07 | |
| fungi | "people"=="lazy americans"? | 17:07 |
| anteaya | nibalizer: did you disappear again? | 17:08 |
| mtreinish | fungi: I don't remember who exactly, this was well over a year ago. It might have been a lazy australian too... :) | 17:08 |
| fungi | i would have accommodated that request by starting to list meetings in both utc and tahiti time | 17:09 |
| anteaya | ha ha ha | 17:09 |
| anteaya | they have their own time zone | 17:09 |
| fungi | "TAHT (Tahiti Time) UTC-10:00" | 17:09 |
| anteaya | yes, I am seeing that | 17:10 |
| anteaya | -10 is a nice round number | 17:10 |
| *** aysyd has joined #openstack-infra | 17:10 | |
| clarkb | fungi: isnt there a -13 timezone too or something | 17:10 |
| clarkb | they are an hour off but want to be in the same "day" as AU or something | 17:10 |
| anteaya | and they don't observe daylight savings time | 17:10 |
| anteaya | so time is consistent in Tahiti | 17:11 |
| *** gokrokve has joined #openstack-infra | 17:11 | |
| clarkb | anteaya: and in arizona | 17:11 |
| anteaya | yay arizona | 17:11 |
| anteaya | and Saskatchewan | 17:11 |
| anteaya | time changes upset the cows | 17:11 |
| *** r-daneel has quit IRC | 17:11 | |
| clarkb | moo | 17:12 |
| fungi | clarkb: nah, i just picked tahiti because it allows me to imagine sitting on a polynesian beach that way | 17:12 |
| *** pc_m_ has joined #openstack-infra | 17:12 | |
| mtreinish | fungi: ok it looks like the patch fixed the puppetboard failure on logstash.o.o if you want to kick off the worker again | 17:12 |
| *** ashaeron has quit IRC | 17:12 | |
| mmedvede | I have a question with this https://review.openstack.org/#/c/120940/. It got merged and breaks devstack on Fedora 20. | 17:12 |
| fungi | mtreinish: yep, checking in on it now | 17:12 |
| anteaya | not as much fun as imaging sitting in a snow bank in Saskatchewan, eh? | 17:13 |
| clarkb | mmedvede: interesting that it broke f20 since it appears to just be setting config options | 17:13 |
| clarkb | mmedvede: what is the question? | 17:13 |
| nibalizer | anteaya: no i was just poking the infra agenda | 17:14 |
| anteaya | mmedvede: well ianw was your sole core and approval on that one | 17:14 |
| *** jlibosva has quit IRC | 17:14 | |
| nibalizer | what about the blacksmith gem? | 17:14 |
| clarkb | anteaya: that issue was breaking the gate I think it got approved quickly | 17:14 |
| anteaya | nibalizer: cool, so the patch that summons the blacksmith gem | 17:14 |
| mmedvede | clarkb: I guess the question is, shouldn't f20 gate be voting? | 17:14 |
| anteaya | nibalizer: have you used that gem before successfully? | 17:14 |
| nibalizer | yes | 17:15 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Filter out some Tempest tests for Nova cells https://review.openstack.org/135064 | 17:15 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Remove -full designation from cells job https://review.openstack.org/135360 | 17:15 |
| clarkb | mmedvede: I think it was at one point but was too unreliable so it was unvoting | 17:15 |
| clarkb | mmedvede: iirc issues with firewalld or something silly | 17:15 |
| clarkb | because fedora | 17:15 |
| anteaya | nibalizer: great, so now help me understand why using an unsecured node is acceptable | 17:15 |
| nibalizer | i uploaded one of my personal modules to the forge using those exact commands | 17:15 |
| fungi | mmedvede: you tracked that change down as what broke testing on f20, or merely observed that the f20 job was failing on that change? | 17:15 |
| anteaya | nibalizer: since that appears to be part of the workflow accourding to the commit message | 17:15 |
| anteaya | nibalizer: awesome, thanks | 17:15 |
| fungi | clarkb: it was a speed/performance issue with firewalld iirc | 17:15 |
| nibalizer | oh the intention was to use only a secure node | 17:15 |
| mmedvede | fungi: we use f20 as well, and have the same error | 17:15 |
| clarkb | fungi: ya with libvirt | 17:16 |
| fungi | mmedvede: and f20 was working up until that change merged? | 17:16 |
| anteaya | nibalizer: okay so in the series of commands in that patch, we will only be using a secure node? | 17:16 |
| nibalizer | ya thats what node: pypi means | 17:16 |
| anteaya | nibalizer: can you rework the commit message then please | 17:16 |
| nibalizer | in some future where we make everything perfect we would probably rename pypi.slave.o.o to trusted.slave.o.o or something | 17:17 |
| sdague | alaski: +2 | 17:17 |
| anteaya | since I read that the workflow required the use of an unsecured node | 17:17 |
| mmedvede | fungi: unfortunately we were down before this change merged, so can not confirm. But the error clearly points to that commit | 17:17 |
| anteaya | nibalizer: ah, that might also be a useful bit of info for the commit message or a comment on the patch | 17:17 |
| *** baoli has quit IRC | 17:17 | |
| *** pc_m_ has quit IRC | 17:17 | |
| fungi | mmedvede: "/opt/stack/new/devstack/lib/horizon: line 145: django-admin.py: command not found" | 17:18 |
| nibalizer | anteaya: okay i'll take a stab at it and ask you to look again | 17:18 |
| mmedvede | fungi: the devstack error being: /opt/stack/new/devstack/lib/horizon: line 145: django-admin.py: command not found | 17:18 |
| fungi | mmedvede: where does that reside on f20? | 17:18 |
| anteaya | nibalizer: thank you | 17:18 |
| *** mudassirlatif has joined #openstack-infra | 17:18 | |
| anteaya | nibalizer: the patch itself looks fine, the communciation around it left me with questions, which you have answered | 17:19 |
| *** dustins has quit IRC | 17:19 | |
| anteaya | so an updated commit message should fix my understanding | 17:19 |
| nibalizer | anteaya: sure | 17:20 |
| *** dkranz has quit IRC | 17:20 | |
| nibalizer | do you think i should rip out that stuff about the pypi jobs? | 17:20 |
| alaski | sdague: nice, thanks | 17:20 |
| nibalizer | is taht part of whats making it confusing? | 17:20 |
| mmedvede | fungi: not sure I get the question. Looking through devstack, 'django-admin.py' is not used anywhere, except for the lib/horizon, and was added with today's merge | 17:20 |
| anteaya | nibalizer: the mention of a workflow using an unsecure node confused me | 17:21 |
| nibalizer | okay yea ill rip it out then | 17:21 |
| anteaya | if the purpose of the pypi node is that it is a trusted node (which I didn't know previously) then that is useful | 17:21 |
| fungi | mmedvede: i'm going to guess it's provided by a django-related package which we're installing on ubuntu but not on fedora | 17:21 |
| fungi | ianw: ^ any ideas there? | 17:21 |
| fungi | alternatively, it may just not be in the executable path | 17:22 |
| *** jpich has quit IRC | 17:22 | |
| clarkb | fungi: do you know off the top of your head if our python bugs in trusty have moved since the summit? | 17:22 |
| clarkb | https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907 :/ doesn't appear to have moved | 17:23 |
| uvirtbot | Launchpad bug 1367907 in python3.4 "Segfault in gc with cyclic trash" [High,Fix released] | 17:23 |
| fungi | clarkb: no updates in any of the ubuntu bugs, though glanceclient fixed up theirs | 17:23 |
| *** dtalton has joined #openstack-infra | 17:24 | |
| mmedvede | fungi: ah, ok. I would look deeper into that. thanks | 17:24 |
| krtaylor | mmedvede, we need to figure out how to take this back to the third-party CI monitoring discussion, a patch shouldn't be merged on failure of a valid CI system | 17:25 |
| fungi | mmedvede: based on my reading, django apps would normally provide their own admin.py file | 17:25 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add job to publish puppet modules to the forge https://review.openstack.org/134834 | 17:25 |
| clarkb | krtaylor: is that re the f20 test? | 17:25 |
| fungi | krtaylor: the f20 job is non-voting because it's still unreliable | 17:25 |
| clarkb | krtaylor: it was run upstream not third party and was deemed unreliable enough to make not voting. I don't think there was anything wrong with merging that change | 17:25 |
| clarkb | particularly since it fixed a real issue in horizons devstack deployments | 17:25 |
| krtaylor | clarkb, yes, it would have helped Ian if he knew that f20 was working properly | 17:26 |
| clarkb | krtaylor: ian does know | 17:26 |
| *** Ryan_Lane has joined #openstack-infra | 17:26 | |
| fungi | i think he was the one who added that job even | 17:26 |
| krtaylor | clarkb, right third-party is where it is being worked, not that it was third-party | 17:26 |
| clarkb | (ianw has done most of the work around running devstack on f20 and centos7. In this particular case the gate was completely broken because horizon and their css compilation stuff. We fixed it) | 17:26 |
| *** Ryan_Lane has quit IRC | 17:27 | |
| clarkb | fixing it was at the cost of an already unreliable test | 17:27 |
| clarkb | its a reasonable trade off | 17:27 |
| krtaylor | clarkb, yes, I am thinking in general, it would be good to indicate that a system was "trusted" or working properly | 17:27 |
| *** Ryan_Lane has joined #openstack-infra | 17:28 | |
| clarkb | fungi: I have just realized that I am basically on an east coast shcedule. I am ready for lunch right now | 17:28 |
| sdague | clarkb: nice | 17:28 |
| *** unicell has quit IRC | 17:29 | |
| *** bdpayne has joined #openstack-infra | 17:29 | |
| *** shashankhegde has quit IRC | 17:29 | |
| *** amotoki_ has quit IRC | 17:29 | |
| krtaylor | clarkb, something spicy? :) | 17:29 |
| jerryz | clarkb: have you seen zmq event publisher stop publishing events to nodepool? the zmq plugin on my jenkins somehow didn't work properly again and i had to restart jenkins. i also noticed that 8888 port won't be opened until a job started running, and from then on, 8888 stays open | 17:30 |
| *** vigneshvar has joined #openstack-infra | 17:30 | |
| anteaya | clarkb: well getting closer to home | 17:30 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Initial import of puppetlabs-apache 0.0.4 https://review.openstack.org/135369 | 17:30 |
| clarkb | jerryz: its normal that it won't open the socket until a job runs | 17:30 |
| *** yfried has joined #openstack-infra | 17:31 | |
| clarkb | jerryz: since 0mq is supposed to make connecting transparent that shouldn't be an issue. That said yes, we have seen clients be unable to reconnect if they are unconnected for extended periods of time | 17:31 |
| clarkb | I think its a bug in pyzmq | 17:31 |
| clarkb | in our case restarting the clients makes them happy again | 17:31 |
| jerryz | clarkb: but nodepool might miss the first event | 17:31 |
| clarkb | jerryz: yes, but that is true regardless | 17:32 |
| *** dustins has joined #openstack-infra | 17:32 | |
| jerryz | clarkb: if the port only open on a job started | 17:32 |
| clarkb | (I would not be opposed to opening the socket on jenkins start up. there was a reason that it was easier not to that I don't remember offhand (would need to read git log and code)) | 17:33 |
| *** vigneshvar_ has joined #openstack-infra | 17:33 | |
| fungi | nodepool can also miss events if there's a network disruption causing substantial packet loss, so workarounds are necessary sometimes regardless | 17:33 |
| clarkb | right its racy regardless | 17:33 |
| clarkb | so I didn't worry about it a whole lot | 17:34 |
| clarkb | (and nodepool does its internal cleanups too so it should be fine long term) | 17:34 |
| jerryz | clarkb: but nodepool need to see a onStart? event to set the node to be 'used' | 17:35 |
| clarkb | jerryz: it doesn't absolutely need to | 17:35 |
| jerryz | clarkb: if nodepool missed that event and connects zmq later, that node would be still 'ready'? | 17:35 |
| openstackgerrit | Ben Nemec proposed openstack-dev/hacking: Enforce import group ordering https://review.openstack.org/54403 | 17:35 |
| clarkb | jenkins will offline the node, then 8 hour later or whenever you have configured nodepool to cleanup the node will be deleted | 17:35 |
| clarkb | jerryz: and if nodepool gets the finished event it goes straight to delete | 17:36 |
| jerryz | clarkb: it is not my case | 17:36 |
| jerryz | clarkb: maybe my nodepool is too old? | 17:36 |
| clarkb | mabe? I don't know how old your nodepool is :) | 17:36 |
| *** vigneshvar has quit IRC | 17:37 | |
| *** sarob has quit IRC | 17:37 | |
| sdague | fungi / anteaya / clarkb - https://review.openstack.org/#/c/135064 and the dependant patch will let the nova cells folks start making forward progress | 17:37 |
| *** sarob has joined #openstack-infra | 17:37 | |
| jerryz | clarkb: Feb, 2014 | 17:37 |
| clarkb | krotscheck: when you wake up I am ready to start merging storyboard use in infra docs changes. But wanted to make sure I wasn't doing anything wrong | 17:37 |
| JayF | Would love to get the final +2A on this: https://review.openstack.org/#/c/134438/ as we finally got the tests to pass. | 17:37 |
| clarkb | krtaylor: er doing anything wrong by moving forward on that | 17:38 |
| clarkb | jerryz: that is pretty old. But I don't think that is the issue since the gearman plugin is older than that iirc | 17:38 |
| *** links has quit IRC | 17:38 | |
| clarkb | jerryz: in any case if you want to fix the gearman plugin to be a bit more statefully correct that is fine | 17:38 |
| clarkb | its just never been an issue for us | 17:38 |
| *** zul has quit IRC | 17:38 | |
| openstackgerrit | Steven Weston proposed openstack-infra/infra-specs: Third Party CI Dashboard Spec https://review.openstack.org/135170 | 17:39 |
| notmyname | anyone heard reports of gerrit logging in not working in safari? (yes I use safari for reasons) seems that something has changed in OS X, gerrit, or ubuntu one openid in the week I've been out | 17:39 |
| *** yamamoto has joined #openstack-infra | 17:39 | |
| clarkb | zaro: krotscheck annegentle ^ | 17:39 |
| sdague | JayF: +A | 17:40 |
| *** jamespage_ has joined #openstack-infra | 17:40 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Adding readme https://review.openstack.org/135376 | 17:40 |
| JayF | sdague: awesome, tyvm | 17:40 |
| jerryz | clarkb: what i saw on this 'zmq not publishing event' issue is nodepool's slaves stays ready however there were jobs on them, and the node didn't get cleaned up, leaving stale configurations making the next build on it un-trustworthy | 17:41 |
| jerryz | clarkb: also offline nodes got piled up on jenkins | 17:41 |
| *** bhuvan has joined #openstack-infra | 17:41 | |
| clarkb | jerryz: oh are you not using single use nodes? | 17:42 |
| sweston | anteaya: when you have a moment, https://review.openstack.org/135170 does this look better? | 17:42 |
| *** esker has joined #openstack-infra | 17:42 | |
| clarkb | jerryz: I would check that you got the onfinished events | 17:42 |
| jerryz | clarkb: some jobs used it. hence offline nodes on jenkins | 17:42 |
| clarkb | jerryz: if you get those nodepool should set to delete | 17:42 |
| dims | clarkb, fungi: status.json does not have "account" information, is it easy to add? | 17:43 |
| *** gfidente has joined #openstack-infra | 17:43 | |
| jerryz | clarkb: but those are jot offline, but they were still in ready status on nodepool | 17:43 |
| *** esker has quit IRC | 17:43 | |
| clarkb | if ou don't get those then things will pile up. worst case is nodepool will wait until cleanup time and delete nodes then | 17:43 |
| clarkb | jerryz: yes but are you getting the onfinished event? | 17:43 |
| *** s0nea has quit IRC | 17:43 | |
| *** teran_ has quit IRC | 17:43 | |
| clarkb | if you are not getting the event at all that is a separate issue | 17:43 |
| *** esker has joined #openstack-infra | 17:43 | |
| jerryz | clarkb: no events at all | 17:44 |
| anteaya | sweston: ugly red rectangles gone and something for a licence header, thank you | 17:44 |
| clarkb | jerryz: ok I think that is a completely different issue and isn't related to the socket being opened late | 17:44 |
| *** yamamoto has quit IRC | 17:44 | |
| sweston | anteaya: np, thanks for your review | 17:44 |
| anteaya | sweston: I'm uncertain if lines 10-14 stay, perhaps you can look at some of the merged specs? | 17:44 |
| clarkb | jerryz: instead I think that is the pyzmq bug where reconnections don't work after a prolonged period | 17:44 |
| jerryz | clarkb: yeah. that why i restarted jenkins | 17:44 |
| clarkb | jerryz: its client side not server side aiui | 17:45 |
| *** baoli has joined #openstack-infra | 17:45 | |
| fungi | dims: are you inquiring about having zuul report account information from gerrit? | 17:45 |
| clarkb | jerryz: restarting nodepool should fix it | 17:45 |
| fungi | dims: and if so, which account(s)? | 17:45 |
| clarkb | though maybe its corrupt session state on one side and restarting any side clears it up? | 17:45 |
| *** harlowja_away is now known as harlowja | 17:45 | |
| clarkb | we could try building the plugin with a newer jeromq | 17:46 |
| jerryz | clarkb: next time i will use the zmq-stream.py to see which side is bad | 17:46 |
| *** otherwiseguy has quit IRC | 17:47 | |
| *** s0nea has joined #openstack-infra | 17:47 | |
| *** KanagarajM has quit IRC | 17:47 | |
| dims | fungi: working back from the zuul status page (http://status.openstack.org/zuul/), it pulls information from status.json served by zuul, and working back i saw this https://github.com/openstack-infra/zuul/blob/master/zuul/model.py#L951 | 17:47 |
| *** derekh has quit IRC | 17:47 | |
| gfidente | guys, pardon the interruption, I wanted to check one of the two files which have a '@' in their name linked by a CI run at: http://logs.openstack.org/16/135316/1/check-tripleo/check-tripleo-ironic-overcloud-precise-ha/9b8fc6d/logs/overcloud-controller0_logs/mnt/rabbitmq/ , following link doesn't work though, it says File not found | 17:47 |
| *** Ryan_Lane has quit IRC | 17:48 | |
| gfidente | I tried replacing the @ in the link with %40 but that didn't work either | 17:48 |
| dims | fungi: basically trying to see if i can search by name of the person on the zull status page | 17:48 |
| gfidente | can anyone help me get that file somehow? | 17:48 |
| clarkb | gfidente: ya I can rename it for you | 17:48 |
| clarkb | mtreinish: ^ weren't you going to rename those files | 17:48 |
| gfidente | clarkb, ah thanks, are there other options though? | 17:48 |
| clarkb | or hrm I may have written a change to do it that never merged or ? | 17:48 |
| clarkb | gfidente: nope, we debugged this at some point and decided meh | 17:48 |
| *** teran has joined #openstack-infra | 17:48 | |
| clarkb | gfidente: and I thought we had fixed it | 17:48 |
| gfidente | so we better try to rename those early on:) | 17:49 |
| openstackgerrit | Merged openstack-infra/project-config: Remove -full designation from cells job https://review.openstack.org/135360 | 17:49 |
| jerryz | clarkb: another issue i saw was build for old patch didn't get aborted after new patch came in | 17:49 |
| fungi | dims: oh, maybe... that is specifically used for identifying accounts which leave comments so that zuul can treat some users' comments differently from those of other users | 17:49 |
| *** andreykurilin_ has quit IRC | 17:49 | |
| clarkb | I6daaf0475b1dab6bcd56c4d838f4600271d6a8be | 17:49 |
| clarkb | gfidente: ^ | 17:49 |
| dims | fungi: ah ok | 17:49 |
| fungi | dims: the account there is associated with a trigger event, not a changeish | 17:50 |
| dims | fungi: ah ok | 17:50 |
| fungi | dims: and you're probably more interested in searching for something like changes by gerrit owner id | 17:50 |
| clarkb | gfidente: the issue is the path I think | 17:50 |
| dims | fungi: yep | 17:50 |
| clarkb | because tropleo is special | 17:50 |
| * clarkb does a rage dance | 17:50 | |
| gfidente | well we can set the log filename | 17:50 |
| clarkb | SpamapS: ^ can you not be so speial | 17:50 |
| gfidente | to somethine else | 17:50 |
| clarkb | gfidente: thats what my change attempts to do | 17:51 |
| gfidente | using @ is the default though | 17:51 |
| clarkb | gfidente: but tripleo is specail | 17:51 |
| clarkb | and puts things in funny places | 17:51 |
| fungi | dims: so no plumbing there for it, but when zuul queries gerrit for changeish metadata the change owner is reported in teh result, and could in theory be tacked onto the changeish object and exposed in a status view | 17:51 |
| clarkb | like mnt/rabbitmq | 17:51 |
| *** BobBall is now known as BobBall_AWOL | 17:51 | |
| mtreinish | clarkb: rename which files? | 17:51 |
| *** thedodd has joined #openstack-infra | 17:51 | |
| dims | fungi: k, will poke at it | 17:51 |
| fungi | mtreinish: logfiles with @ in their filenames | 17:52 |
| mtreinish | clarkb: oh the rabbit stuff, did I sign up for that, I thought you were going to push the patch | 17:52 |
| gfidente | clarkb, var/log/rabbitmq is a link to mnt/state/var/log/rabbitmq though ... | 17:52 |
| mtreinish | there's never really anything useful in those anyway... | 17:52 |
| clarkb | mtreinish: I did push that patch | 17:52 |
| clarkb | mtreinish: it merged. issue here is tripleo is different | 17:52 |
| mtreinish | clarkb: ah, ok | 17:53 |
| openstackgerrit | Steven Weston proposed openstack-infra/infra-specs: Third Party CI Dashboard Spec https://review.openstack.org/135170 | 17:53 |
| clarkb | gfidente: files have been renamed | 17:53 |
| *** erikwilson has quit IRC | 17:53 | |
| clarkb | gfidente: I am just frustrated that tripleo has in numerous occasions reinvented the wheel slightly differently which means that things never work quite right in the existing CI infrastructure | 17:53 |
| clarkb | so things work until they dont | 17:53 |
| gfidente | clarkb, I think this is indeed renaming the files as it should, but not for the overcloud nodes | 17:54 |
| *** mpaolino has quit IRC | 17:54 | |
| sweston | anteaya: I did look at that, but only the other proposed specs. I looked at a merged spec, and removed the offending line | 17:54 |
| mordred | clarkb: did we ever get anywhere with tchaypo and a plan for solving his use case with zuul? | 17:55 |
| clarkb | mordred: he played with it and I think he decided it was possible | 17:55 |
| mordred | cool | 17:55 |
| clarkb | but not sure if he wanted to take the time (note I had thought he was using this thing in the gate but is apparently only using it for local cloud deployments) | 17:55 |
| anteaya | sweston: thanks | 17:55 |
| openstackgerrit | Merged openstack-infra/project-config: Filter out some Tempest tests for Nova cells https://review.openstack.org/135064 | 17:55 |
| clarkb | so I became less worried (my major concern was replacing zuul in the gate) | 17:55 |
| sweston | anteaya: np, thanks for the help!! | 17:56 |
| gfidente | clarkb, so I am not sure how the devstack_gate stuff is used for the tripleo CI | 17:56 |
| gfidente | tripleo is deploying the cloud using tripleo itself not devstack | 17:56 |
| mordred | gfidente: devstack-gate is the thing taht understands how to get all of the repos into the appropriate states | 17:57 |
| nibalizer | mordred: clarkb i think he is hacking on it | 17:57 |
| mordred | amongst other things | 17:57 |
| mordred | nibalizer: ++ | 17:57 |
| nibalizer | he wanted a new zuul release though | 17:57 |
| *** mmaglana has joined #openstack-infra | 17:57 | |
| *** yjiang5_away is now known as yjiang5 | 17:57 | |
| nibalizer | since its been a few years | 17:57 |
| clarkb | nibalizer: I don't disagree we should release zuul at some point (I defer to jeblair) but its funny that the CD proponents need that :) | 17:57 |
| *** talluri has joined #openstack-infra | 17:58 | |
| mordred | nibalizer: well, the thing you asked about on the mailing list I think is ultimately the same technical problem as tchaypo was looking at | 17:58 |
| mordred | nibalizer: althoguh possbily different interfaces | 17:58 |
| nibalizer | mordred: hrm yea | 17:58 |
| *** emagana has joined #openstack-infra | 17:59 | |
| tchaypo | Agree | 17:59 |
| *** Sincler has joined #openstack-infra | 17:59 | |
| *** teran has quit IRC | 17:59 | |
| gfidente | tchaypo, nice to see you :p | 18:00 |
| *** teran has joined #openstack-infra | 18:00 | |
| tchaypo | From what I've seen it looks like it's possible to use zuul_merger to do what I need | 18:00 |
| tchaypo | I was looking at it today and it doesn't look too difficult | 18:01 |
| *** shashankhegde has joined #openstack-infra | 18:01 | |
| *** vrutkovs has quit IRC | 18:01 | |
| krotscheck | clarkb: All good | 18:02 |
| krotscheck | clarkb: Was offline because I had to move my desk. | 18:02 |
| clarkb | krotscheck: cool /me approves some things | 18:02 |
| tchaypo | I'm guessing the thing you are doing is having some private patches you want to maintain? | 18:02 |
| *** amuller__ has joined #openstack-infra | 18:02 | |
| *** melwitt has joined #openstack-infra | 18:03 | |
| *** mpaolino has joined #openstack-infra | 18:03 | |
| *** Guest32278 is now known as mgagne | 18:03 | |
| fungi | krotscheck: you were going to draft our infra storyboard cut-over announcement right? | 18:03 |
| *** mgagne has quit IRC | 18:03 | |
| *** mgagne has joined #openstack-infra | 18:03 | |
| krotscheck | fungi: That is correct | 18:03 |
| *** gokrokve has quit IRC | 18:03 | |
| *** gokrokve has joined #openstack-infra | 18:03 | |
| *** yfried is now known as yfried|afk | 18:04 | |
| *** vipul has quit IRC | 18:04 | |
| *** jamielennox|away is now known as jamielennox | 18:04 | |
| fungi | okay, cool. i think we want to have consensus around the announcement before i do one last import pass and then immediately lock lp bugs for the relevant projects | 18:04 |
| openstackgerrit | Merged openstack-infra/nose-html-output: Switched documentation to point to storyboard. https://review.openstack.org/133852 | 18:04 |
| mordred | krotscheck: do you mean you had to relocate to a different desk? or you were asked to physically move your current desk object | 18:04 |
| SpamapS | clarkb: what do you want unspecialed? | 18:04 |
| *** jamielennox is now known as jamielennox|away | 18:04 | |
| krotscheck | mordred: Long story. Short version: Too many hires in seattle, no more room, I got kicked out while they built more desks, now I’m back and I have a new one, but I had to move the old stuff. | 18:05 |
| *** bdpayne_ has joined #openstack-infra | 18:05 | |
| clarkb | SpamapS: the most recent thing unrelated to rabbitmq logs was how code is checked out for testing | 18:05 |
| mordred | krotscheck: lovely | 18:05 |
| clarkb | SpamapS: its like devstack gate plus | 18:05 |
| krotscheck | mordred: Ehn. Same desk, different distance to the bathroom. | 18:05 |
| *** jamielennox|away is now known as jamielennox | 18:05 | |
| SpamapS | clarkb: It's like production-oriented :) | 18:05 |
| *** erikwilson has joined #openstack-infra | 18:05 | |
| *** vipul has joined #openstack-infra | 18:05 | |
| krotscheck | Also, closer to the UX/UI/Horizon team now. | 18:05 |
| clarkb | SpamapS: but having consistent log paths so that things like @s in filenames can be dealt with would be nice too | 18:05 |
| SpamapS | clarkb: so yes, devstack-gate plus "the way we want to do it in production" :) | 18:06 |
| clarkb | SpamapS: sure thats fine, maybe we make d-g understand how to do that then | 18:06 |
| clarkb | SpamapS: otherwise its just confusing | 18:06 |
| mordred | ++ | 18:06 |
| clarkb | and people look at me to fix it and I go "lolno" | 18:06 |
| SpamapS | clarkb: we're centralizing on syslog now I believe. | 18:06 |
| *** bdpayne has quit IRC | 18:06 | |
| clarkb | because I have no idea what is going on | 18:06 |
| SpamapS | though I don't know why we did it with logger | 18:06 |
| *** yfried|afk is now known as yfried | 18:07 | |
| * SpamapS needs to go back and fix that to do that with log configuration instead. | 18:07 | |
| mordred | SpamapS: I think the thing is that we'd like for the surrounding infrastructure to be able to do the work of checking out specific changes in locations - so if the CI infrastructure does not know how to prepare git repos in the way you need, we should figure out what you need that we don't already do and add that feature | 18:07 |
| *** mpaolino has quit IRC | 18:07 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:07 | |
| *** zul has joined #openstack-infra | 18:07 | |
| SpamapS | mordred: I thought it did actually. | 18:07 |
| clarkb | devstack gate runs and does a good chunk of the work | 18:08 |
| clarkb | then you run a thing that can completely undo what devstack gate just did | 18:08 |
| *** mpaolino has joined #openstack-infra | 18:08 | |
| SpamapS | like revert things that break our CI, or something else? | 18:08 |
| SpamapS | because until we're in the gate we need that. :-/ | 18:08 |
| clarkb | like checkout not the thing under review | 18:08 |
| clarkb | because this happened for a long time | 18:08 |
| clarkb | with several tripleo projects | 18:09 |
| openstackgerrit | Merged openstack-infra/devstack-gate: multi-node dg setup https://review.openstack.org/106043 | 18:09 |
| SpamapS | ah was that just a misunderstanding maybe? | 18:09 |
| clarkb | sure, but I was asked to help debug and I had no idea where to start | 18:09 |
| SpamapS | I don't really know what we're talking about.. probably need to dive a bit deeper into it to understand. | 18:09 |
| clarkb | and anyone else working on openstack would be confused too | 18:09 |
| *** achanda has joined #openstack-infra | 18:09 | |
| *** _shaps_ has quit IRC | 18:09 | |
| *** luqas has quit IRC | 18:09 | |
| fungi | i think it was more of a "this is mostly a solved problem, let's finish solving it where it's already mostly solved rather than making something new which gets to experience the same bugs and design flaws we've already fixed in the existing thing" | 18:10 |
| SpamapS | clarkb: what _exactly_ happens after devstack-gate runs? | 18:10 |
| clarkb | SpamapS: I have no idea and its part of the problem | 18:10 |
| mordred | fungi: ++ | 18:10 |
| clarkb | but it isn't "run tests" | 18:11 |
| *** thedodd has quit IRC | 18:11 | |
| clarkb | its more along the lines of "do that thing d-g just did so that we can make changes if we want to" | 18:11 |
| clarkb | then run tests | 18:11 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:11 | |
| *** otherwiseguy has joined #openstack-infra | 18:11 | |
| SpamapS | I'm still confused. | 18:12 |
| clarkb | and it is this step I thought that tchaypo was replacing with a new tool | 18:13 |
| clarkb | which is why I suggested the zuul route | 18:13 |
| SpamapS | clarkb: AHHH ok ok ok | 18:13 |
| tchaypo | It sounds like it needs to happen, but no :) | 18:13 |
| openstackgerrit | Merged openstack-infra/gear: Switched documentation to point to storyboard. https://review.openstack.org/133843 | 18:13 |
| tchaypo | That said, the zuul route looks sensible anyway | 18:13 |
| *** cpowell has quit IRC | 18:14 | |
| fungi | so the separate suggestion is to refactor more of toci_gate_test.sh out into devstack-gate, yeah? | 18:14 |
| openstackgerrit | Merged openstack-infra/gearman-plugin: Switched documentation to point to storyboard. https://review.openstack.org/133844 | 18:14 |
| openstackgerrit | Merged openstack-infra/gearman-plugin: Switched documentation to point to storyboard. https://review.openstack.org/133844 | 18:14 |
| clarkb | fungi: or have them use zuul instead which is what my initial thoughts were | 18:14 |
| openstackgerrit | Merged openstack-infra/gerritlib: Switched documentation to point to storyboard https://review.openstack.org/133846 | 18:14 |
| clarkb | fungi: because addingnew features makes more sense in zuul going forward | 18:14 |
| clarkb | and I am almost certain we need new features to support them (and thats fine) | 18:14 |
| openstackgerrit | Merged openstack-infra/gerritbot: Switched documentation to point to storyboard. https://review.openstack.org/133845 | 18:15 |
| openstackgerrit | Merged openstack-infra/jeepyb: Switched documentation to point to storyboard. https://review.openstack.org/133850 | 18:15 |
| fungi | clarkb: oh, you mean from a worker interaction perspective | 18:15 |
| clarkb | yup | 18:15 |
| mordred | and I think there are other people who could use that feature too | 18:15 |
| openstackgerrit | Merged openstack-infra/system-config: Switched documentation to point to storyboard. https://review.openstack.org/133864 | 18:15 |
| fungi | mtreinish: sorry, got sidetracked... i'm looking on logstash.o.o now and can't see any initscript for the subunit gearman worker... how is it supposed to be getting started? | 18:17 |
| tchaypo | Zuul_merger looks like it's already close to what we need. The only caveat I have is that if we start telling people to use it as a simple tool to help build their cloud, it's a hassle to have to tell them to go check | 18:17 |
| *** yfried is now known as yfried|afk | 18:17 | |
| tchaypo | Out zuul from git | 18:17 |
| clarkb | tchaypo: ya I agree we should release the thing | 18:17 |
| tchaypo | Pip install zuul would be much simpler | 18:17 |
| mtreinish | fungi: the worker script only runs on subunit01-worker.o.o | 18:17 |
| tchaypo | Apologies for slow responses, on iPhone right now at dinner | 18:18 |
| mtreinish | it was just getting put on logstash.o.o because I moved it's definition to init.pp | 18:18 |
| fungi | mtreinish: oh, got it | 18:18 |
| *** annegent_ has joined #openstack-infra | 18:18 | |
| fungi | mtreinish: so anyway, the subunit-worker01 instance seems broken to the point where i'm not allowed to ssh in. digging into that now | 18:19 |
| *** packet has joined #openstack-infra | 18:19 | |
| fungi | mtreinish: looks like the puppetmaster hasn't been able to ansible into it either | 18:19 |
| mtreinish | fungi: hmm, time to nuke it and start over? | 18:20 |
| fungi | mtreinish: possibly | 18:20 |
| fungi | i'm seeing if i can figure out what went wrong | 18:20 |
| *** patrickeast has joined #openstack-infra | 18:21 | |
| *** jp_at_hp has quit IRC | 18:21 | |
| fungi | 2014-11-18 18:20:37,320 p=12505 u=root | subunit-worker01.openstack.org : ok=0 changed=0 unreachable=1 failed=0 | 18:21 |
| *** dkranz has joined #openstack-infra | 18:21 | |
| fungi | according to ansible | 18:21 |
| *** hdd has joined #openstack-infra | 18:22 | |
| *** gyee has joined #openstack-infra | 18:22 | |
| *** rushiagr is now known as rushiagr_away | 18:22 | |
| mtreinish | fungi: hmm, it pings fine and nmap shows ssh open (for whatever that's worth) | 18:23 |
| fungi | mtreinish: yeah, it prompts me for a password though | 18:23 |
| fungi | prompts root for one when connecting from puppetmaster too | 18:23 |
| mordred | that's not the thing we really want :) | 18:23 |
| fungi | so i think keys didn't get set up correctly for whatever reason | 18:23 |
| mmedvede | clarkb, fungi: the patch to fix f20 devstack issue: https://review.openstack.org/#/c/135385/ | 18:23 |
| fungi | except i saw no errors to that effect when launching the instance | 18:24 |
| *** amcrn has joined #openstack-infra | 18:24 | |
| fungi | mmedvede: you'll probably want to get some devstack core reviewers to have a look at that too | 18:25 |
| fungi | mmedvede: but lgtm | 18:25 |
| *** nelsnelson has quit IRC | 18:26 | |
| mordred | fungi, clarkb: I can't seem to do timezone math right now - how many hours until the infra meeting? | 18:26 |
| fungi | mordred: 0.5 | 18:26 |
| mordred | ok. cool. thanks | 18:26 |
| *** nelsnelson has joined #openstack-infra | 18:27 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/zuul: Add gerrit owner information to status.json https://review.openstack.org/135386 | 18:28 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/devstack-gate: Refactor the hook logic into a reusable function https://review.openstack.org/123578 | 18:28 |
| *** jamespage_ has quit IRC | 18:29 | |
| mmedvede | fungi: awesome, thanks. I'll try to update it with your suggestion. | 18:29 |
| tchaypo | Talking to nibalizer a bit about his problem | 18:29 |
| tchaypo | I think I have spoken to other people who also have similar issues | 18:30 |
| mordred | yah. it's a thing that is a generalizable problem | 18:30 |
| tchaypo | They have a private stack of patches, so they pull in from upstream infrequently and need to get all the upstream projects in a consistent state and then apply their own stack of patches | 18:30 |
| dims | fungi: i think i spotted where in the code i can grab the gerrit owner information. thanks | 18:31 |
| mordred | tchaypo: yup. and the Merge-from-upstream-infrequently part is the part where sequencing those changes doesn't work | 18:31 |
| *** yfried|afk is now known as yfried | 18:31 | |
| asselin | nibalizer, could you change the topic of this to module-split? https://review.openstack.org/#/c/134812/ | 18:33 |
| mordred | asselin: does it not let you do that? | 18:33 |
| openstackgerrit | Merged openstack-infra/infra-specs: Update puppet module git subtree instructions https://review.openstack.org/129768 | 18:34 |
| asselin | mordred, no | 18:34 |
| asselin | I can only change my own | 18:34 |
| mordred | asselin: well, it lets me, so I've just changed it | 18:34 |
| *** links has joined #openstack-infra | 18:34 | |
| asselin | mordred, you must be core :) | 18:34 |
| asselin | thanks | 18:34 |
| clarkb | so I think `git merge` solves that problem | 18:34 |
| clarkb | but no one wants to do it that way for other raisins | 18:35 |
| *** kirshil has quit IRC | 18:35 | |
| sdague | mmedvede: +2 | 18:35 |
| clarkb | we do this upstream partially for the tags even | 18:35 |
| mordred | clarkb: there are two explicit things that "git merge" is not an answer for here | 18:35 |
| mordred | clarkb: one is "I want to merge upstream changes to nova into my local repo" - git merge works great for that ... | 18:36 |
| mmedvede | sdague: yay, thanks | 18:36 |
| mordred | clarkb: but if you have not merged all of openstack for say, 3 days, it's likely that you'll also need keystone changes for that merge to work | 18:36 |
| openstackgerrit | Merged openstack-infra/infra-specs: Add commit msg requirements infra #agreed to https://review.openstack.org/132129 | 18:36 |
| mordred | clarkb: so the strict sequencing requirements that are usually in place for something to pass an integrated gate are impossible to meet | 18:36 |
| tchaypo | Git merge works | 18:36 |
| tchaypo | Git merge is exactly the right thing to do | 18:37 |
| *** zul has quit IRC | 18:37 | |
| tchaypo | Doing it regularly, in the right sequence, in the same way, requires a script | 18:37 |
| *** talluri has quit IRC | 18:38 | |
| clarkb | mordred you could do intermediate "checkpoint" merges using openstack/openstack for order info | 18:38 |
| mordred | clarkb: to what benefit? | 18:38 |
| *** gondoi is now known as zz_gondoi | 18:38 | |
| *** Ryan_Lane has joined #openstack-infra | 18:39 | |
| *** jgallard has quit IRC | 18:39 | |
| clarkb | so that nova doesnt merge things without the needed keystone commits | 18:40 |
| *** cpowell has joined #openstack-infra | 18:40 | |
| tchaypo | So that a downstream who want to get all the tools checked out in a consistent state | 18:40 |
| *** achanda has quit IRC | 18:41 | |
| mordred | clarkb: but that has no value in the downstream context - what they want to check is "is the result of merging up with current state of upstream" goign to work - it's a different usecase than "I am a developer with a patch I want to land" | 18:41 |
| tchaypo | Can check out a checkpoint from openstack/openstack and know that it has a set of commits that passed the gate together? | 18:41 |
| *** annegent_ has quit IRC | 18:41 | |
| clarkb | mordred that gets them working trees | 18:41 |
| clarkb | because you wont get ahead in a single project | 18:41 |
| clarkb | tchaypo exactly | 18:42 |
| *** annegent_ has joined #openstack-infra | 18:42 | |
| mordred | sigh. still missing the point | 18:42 |
| mordred | nope | 18:42 |
| mordred | ok. say we grab the upstream commits from upstraem from openstack/openstack | 18:42 |
| mordred | so that the upstraem commits are all a state taht is known to work together | 18:42 |
| mordred | and now we merge that known good state with teh current known good state internally | 18:42 |
| krotscheck | fungi: https://etherpad.openstack.org/p/storyboard-migration-email | 18:42 |
| mordred | and tehn propose those merges for review for sanity checking | 18:42 |
| mordred | at that point | 18:42 |
| mordred | how are those commits suppose to land | 18:43 |
| mordred | IF | 18:43 |
| tchaypo | Known food upstream + previously working private patches? | 18:43 |
| anteaya | krotscheck: wow is that a green | 18:43 |
| mordred | in the period between the last merge and now a sequence of commits landed upstream | 18:43 |
| tchaypo | s/food/good/ | 18:43 |
| krotscheck | anteaya: Not bright enough for ya? Just a minute... | 18:43 |
| mordred | know good upstream isn't the problem | 18:43 |
| mordred | it's sequencing that has all landed upstream | 18:43 |
| clarkb | mordred that is why you need sequenced che kpoints that work. | 18:43 |
| mordred | so it's hidden in the merge commits | 18:43 |
| *** achanda has joined #openstack-infra | 18:44 | |
| *** talluri has joined #openstack-infra | 18:44 | |
| mordred | what youre suggesting is figuring out, in some way, where sequenced patch landings might have happened upstream | 18:44 |
| mordred | even though that informatino is not encoded anywhere | 18:44 |
| mordred | OR | 18:44 |
| clarkb | which is what I am saying you can build a set of jumps based on who changes when that is > than going one commit at a time | 18:44 |
| clarkb | it is encoded in openstack/openstack | 18:44 |
| mordred | no, it's not | 18:44 |
| mordred | EVERY commit is encoded there | 18:44 |
| clarkb | it is | 18:44 |
| clarkb | yes | 18:44 |
| tchaypo | They way we've been using p-s-r, there are no merge commits preserved | 18:45 |
| mordred | but you do not know which of them isa forced sequnce | 18:45 |
| clarkb | and you can decode this info by walking the tree | 18:45 |
| tchaypo | Every time we run we reset to upstream master then apply patches | 18:45 |
| mordred | how | 18:45 |
| mordred | how can you decide it | 18:45 |
| tchaypo | But we don't attempt to keep all the tools in sync | 18:45 |
| mordred | without actually doing an upstream merge FOR EVERY COMMIT that happened upstraem? | 18:45 |
| clarkb | if 10 nova changes merge without any other changes that is collapseable | 18:45 |
| clarkb | and using jogos graphs you can expand those rules | 18:45 |
| mordred | imagine a week or two weeks' of commits | 18:46 |
| mordred | now imagine you want to merge with upstream | 18:46 |
| *** apmelton_ has quit IRC | 18:46 | |
| mordred | with your plan, you would literally wind up with 1000 different merge commit proposals into a downstream gerrit | 18:46 |
| *** apmelton has joined #openstack-infra | 18:46 | |
| clarkb | to say a series of nova and swift changes are collapseable but neutron is not collapseable with nova | 18:46 |
| *** talluri has quit IRC | 18:46 | |
| anteaya | krotscheck: wow, that is bright | 18:47 |
| clarkb | thats absolute worst case | 18:47 |
| clarkb | we do 500ish commits per week max | 18:47 |
| mordred | ok. so even if it's only 50 merge commit proposals | 18:48 |
| clarkb | I think you can get better particularly since so much dev is in nova | 18:48 |
| mordred | why would you BOTHER | 18:48 |
| mordred | when all you want is to fast forward 18 repos to a new known good state? | 18:48 |
| mordred | like, what benefit does it have, other than making this tool happy | 18:48 |
| clarkb | I am not saying you should just that you can | 18:48 |
| clarkb | nothibg stops you from fast forwarding the world | 18:49 |
| mordred | it does | 18:49 |
| *** pblaho has quit IRC | 18:49 | |
| tchaypo | Mordred - it sounds to me as though you're trying to apply the upstream 18 patches on top of the merge you did two weeks ago? | 18:49 |
| mordred | there is no way to land coordinated patchsets with a zuul managed world | 18:49 |
| clarkb | just have an openstack/openstack thing and fast forward | 18:49 |
| clarkb | mordred ^ would do it | 18:49 |
| mordred | I'm sorry | 18:49 |
| tchaypo | Rather than taking your internal stack and applying them fresh on master? | 18:49 |
| mordred | I used the wrong word | 18:49 |
| mordred | I do not mean git fast forward | 18:49 |
| mordred | tchaypo: no, I'm talking about merging the internal stack | 18:49 |
| *** otter768 has joined #openstack-infra | 18:49 | |
| mordred | I'm just saying that you're going to do it in 18 projects at the same time | 18:49 |
| *** markmcclain has joined #openstack-infra | 18:51 | |
| *** markmcclain has quit IRC | 18:51 | |
| clarkb | I do think solving this in a gated fashion isnt quite right | 18:51 |
| clarkb | this is a super feature atop gating | 18:51 |
| mordred | yes. that is the problem | 18:51 |
| jesusaurus | clarkb: what do you mean by that? | 18:52 |
| clarkb | the assertion is gatibg is wrong | 18:52 |
| mordred | and I think the solution is a thing that can push the combined merges once they've been tested | 18:52 |
| clarkb | so dont gate just move to where you want to be | 18:52 |
| *** markmcclain has joined #openstack-infra | 18:52 | |
| mordred | but to do that, I _think_ that you'd want something like tchaypo's feature to help construct the state that you want to test | 18:52 |
| mordred | so that you can test the merges before you merge them all | 18:52 |
| mordred | but I don't think that the normal gate workflow is the right way to finalize them | 18:52 |
| mordred | (is basically what I've been trying to get at) | 18:53 |
| gfidente | dumb question, if I wanted to put a long link to some logs.openstack.org file in a commit, can I truncate it somehow or shall I avoid it entirely? | 18:53 |
| sdague | mordred: would it make sense to do this with tags ? | 18:53 |
| clarkb | ok I missed that I thought you wanted to gate this stuff | 18:53 |
| mordred | sdague: wouldn't really help I don't think | 18:53 |
| mordred | clarkb: no - gating this is just what the current mechanism is | 18:53 |
| mordred | or, rather | 18:53 |
| mordred | the gate designed for individual developer's new patches | 18:54 |
| mordred | you do want to TEST the result of the new state before you push it | 18:54 |
| mordred | because it's openstack | 18:54 |
| sdague | well, if you rolled a week42 tag across components, and made sure that they all worked at that ref | 18:54 |
| tchaypo | Mordred: p-s-r isn't quite right either, only because it's too focused on pulling patches from Gerrit reviews | 18:54 |
| tchaypo | It sounds like you want something that can manage a set of patches | 18:54 |
| mordred | no. not at all | 18:55 |
| mordred | I want something similar to what sdague just said | 18:55 |
| mordred | EXCEPT | 18:55 |
| mordred | seeing as how that's exactly how zuul tests all changes | 18:55 |
| *** sputnik13 has joined #openstack-infra | 18:55 | |
| clarkb | let me describe how I see this being implemented | 18:55 |
| mordred | I want zuul to be able to know that a given event needs to construct a zuul ref across 18 different projects so that that combinatino can be tested | 18:55 |
| tchaypo | So every time the gate passes we add a tag to every repo with a timestamp? | 18:56 |
| clarkb | you need a new pipeline manager "CoDependentPipelineManager" | 18:56 |
| sputnik13 | hello, we're having issues adding ssh keys to launchpad accounts and having the key be recognized by gerritt | 18:56 |
| mordred | clarkb: yes. exactly | 18:56 |
| clarkb | that works like DependentPipelineManager in 99% of all cases | 18:56 |
| sputnik13 | can anyone help? | 18:56 |
| mordred | tchaypo: nonon. tags totally not needed | 18:56 |
| fungi | gfidente: logs get expired after a few months, so probably best not to put urls for them in long-lived git commits | 18:56 |
| mordred | sputnik13: gerrit does not know anything about launchpad keys | 18:56 |
| *** harlowja is now known as harlowja_away | 18:56 | |
| mordred | sputnik13: you need to add keys to gerrit | 18:56 |
| fungi | sputnik13: gerrit doesn't pull ssh keys from launchpad | 18:56 |
| clarkb | mordred: but when a merge commit is pushed all other merge commits across projects in that manager with some common key are tested and merged atomically if the thing passes | 18:56 |
| gfidente | fungi can I reference the issue with the @ in filenames somehow? | 18:56 |
| jesusaurus | can't we use the openstack/openstack project to get a state of commits across all projects? | 18:57 |
| sputnik13 | mordred: oh really, I thought adding the key to launchpad automatically gets it in to gerrit | 18:57 |
| mordred | clarkb: yes. | 18:57 |
| gfidente | (fungi and also thanks) | 18:57 |
| fungi | gfidente: is there a bug number? | 18:57 |
| mordred | sputnik13: not at all | 18:57 |
| anteaya | sputnik13: where did you get that idea from? | 18:57 |
| mordred | anteaya: it used to | 18:57 |
| gfidente | fungi I don't think so, indeed I am not sure why we have this limitation | 18:57 |
| anteaya | mordred: oh | 18:57 |
| mordred | about 2 years ago I believe | 18:57 |
| anteaya | very old docs floating around still? | 18:57 |
| sputnik13 | anteaya: some misbegotten notion that the two are sync'd apparently :-) | 18:57 |
| clarkb | mordred: which fits into my thinking that this is gating + extra feature | 18:58 |
| mordred | clarkb: exactly | 18:58 |
| anteaya | sputnik13: seems so | 18:58 |
| mordred | clarkb: that is exactly what I've been trying to say this whole time :) | 18:58 |
| anteaya | sputnik13: can I offer you this? http://docs.openstack.org/infra/manual/developers.html#getting-started | 18:58 |
| clarkb | sorry I kept reading this as somehow make this work in current gating schema | 18:58 |
| mordred | nope. | 18:58 |
| fungi | sputnik13: the only current connection between lp and review.openstack.org is the openid metadata used when authenticating in the webui (lp sometimes passes through e-mail address and/or username) | 18:59 |
| mtreinish | clarkb: do you have any thoughts on?: https://review.openstack.org/135161 | 18:59 |
| sputnik13 | fungi: I see, thank you for the clarification | 18:59 |
| mordred | I'm going to be about 5 late to the infra meeting - I need to switch couches | 19:00 |
| anteaya | mordred: safe travels | 19:00 |
| *** mpaolino has quit IRC | 19:00 | |
| clarkb | mtreinish: commented | 19:00 |
| fungi | gfidente: i'm going to guess it's either a limitation in apache autoindex or our log profiler cgi | 19:00 |
| clarkb | its ok keystone is still going | 19:00 |
| *** cpowell has quit IRC | 19:00 | |
| fungi | yeah | 19:00 |
| *** kmartin has quit IRC | 19:01 | |
| sputnik13 | anteaya: thank you, we were actually walking through this but glossed over the "upload an ssh key" part equating that to adding a key to the launchpad account. I realize now the link pionts to the gerritt profile | 19:01 |
| anteaya | great | 19:01 |
| anteaya | glad it is of us | 19:01 |
| gfidente | fungi, I'll reference the other change from clarkb https://review.openstack.org/#/c/122885/ no worries | 19:01 |
| anteaya | hope your group is successful | 19:01 |
| sputnik13 | perhaps the text can be modified to say "upload an ssh key on gerrit" or something to that effect | 19:01 |
| *** Ark has joined #openstack-infra | 19:01 | |
| *** emagana has quit IRC | 19:01 | |
| *** cpowell has joined #openstack-infra | 19:02 | |
| mtreinish | clarkb: ok I'll switch it to https. But IIRC part of the reason for moving things over to our git servers was to use git:// | 19:02 |
| *** Ark is now known as Guest77830 | 19:02 | |
| clarkb | mtreinish: in the many many busy test slaves and things | 19:02 |
| anteaya | sputnik13: well went you get to offering your first patch, after sandbox do offer it as a patch | 19:02 |
| clarkb | mtreinish: the small handful of long lived servers talk to git much less frequently | 19:02 |
| mtreinish | clarkb: ah, ok | 19:02 |
| mtreinish | makes sense | 19:02 |
| anteaya | sputnik13: http://git.openstack.org/cgit/openstack-infra/infra-manual/ | 19:02 |
| anteaya | sputnik13: welcome aboard | 19:03 |
| *** liam__ has joined #openstack-infra | 19:03 | |
| sputnik13 | anteaya: heheh, thanks :-) | 19:03 |
| anteaya | thank you | 19:03 |
| *** emagana has joined #openstack-infra | 19:03 | |
| anteaya | I look forward to reviewing your patch | 19:03 |
| sputnik13 | anteaya: :-) | 19:04 |
| anteaya | :D | 19:04 |
| *** liam__ has quit IRC | 19:05 | |
| *** koolhead17 has quit IRC | 19:08 | |
| *** koolhead17 has joined #openstack-infra | 19:09 | |
| *** Guest77830 has quit IRC | 19:12 | |
| *** kgiusti has joined #openstack-infra | 19:13 | |
| *** koolhead17 has quit IRC | 19:13 | |
| sputnik13 | anteaya: do we need to create a "bug" for manual changes? | 19:15 |
| *** thedodd has joined #openstack-infra | 19:15 | |
| anteaya | no | 19:15 |
| anteaya | just offer the patch | 19:16 |
| sputnik13 | I'm new to the whole openstack contribution and gerrit process | 19:16 |
| anteaya | thanks for asking | 19:16 |
| anteaya | you are doing fine | 19:16 |
| sputnik13 | k :-) | 19:16 |
| anteaya | keep asking questions | 19:16 |
| anteaya | :D | 19:16 |
| anteaya | every program is different about bugs | 19:16 |
| anteaya | in infra we only use bugs when we really have to | 19:16 |
| anteaya | we prefer to just review patches | 19:16 |
| *** photocyte has joined #openstack-infra | 19:17 | |
| openstackgerrit | Merged openstack-infra/storyboard: Switch to "serve" verb, and a minor clean up. https://review.openstack.org/133361 | 19:17 |
| clarkb | mtreinish: sdague jogo e-r graphs page looks stale | 19:18 |
| clarkb | (haven't debugged yet /me goes back to meeting) | 19:18 |
| *** koolhead17 has joined #openstack-infra | 19:18 | |
| sdague | clarkb: yep, it does | 19:18 |
| sdague | probably hung lock | 19:18 |
| openstackgerrit | Min Pae proposed openstack-infra/infra-manual: Adds clarification about uploading ssh key https://review.openstack.org/135397 | 19:18 |
| clarkb | kk I can look after the meeting | 19:18 |
| anteaya | sputnik13: if you have the time this is a presentation about how to contribute to one of our programs called neutron: https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/land-your-first-neutron-patch | 19:18 |
| *** che-arne has quit IRC | 19:19 | |
| anteaya | sputnik13: there is a lot of useful stuff in there that is good for how to contribute to any openstack program | 19:19 |
| *** baoli has quit IRC | 19:19 | |
| sputnik13 | that would be me | 19:19 |
| sputnik13 | https://review.openstack.org/135397 | 19:19 |
| *** koolhead17 has quit IRC | 19:19 | |
| sputnik13 | anteaya: thank you, I will take a look | 19:19 |
| *** koolhead17 has joined #openstack-infra | 19:20 | |
| *** koolhead17 has joined #openstack-infra | 19:20 | |
| anteaya | sputnik13: nice patch | 19:20 |
| *** r-daneel has joined #openstack-infra | 19:20 | |
| *** dtalton has quit IRC | 19:20 | |
| sdague | clarkb: when you do so, we should try to figure out if there are logs on the oldest hung process to figure out what blipped it | 19:21 |
| clarkb | kk | 19:21 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: User token API https://review.openstack.org/134409 | 19:21 |
| *** sputnik13 has quit IRC | 19:21 | |
| *** sputnik13 has joined #openstack-infra | 19:23 | |
| sputnik13 | anteaya: thank you :-) | 19:23 |
| *** photocyte is now known as dtalton | 19:23 | |
| anteaya | thank you! | 19:23 |
| *** otherwiseguy has quit IRC | 19:23 | |
| *** koolhead17 has quit IRC | 19:24 | |
| anteaya | sputnik13: what programs do you want to contribute to? | 19:24 |
| *** annegent_ has quit IRC | 19:25 | |
| sputnik13 | anteaya: I recently started at HP, so I will be working on some of the projects we have here, but generally I have a lot of interest in networking, security, and management systems, those are what I've worked on most of my career | 19:26 |
| *** koolhead17 has joined #openstack-infra | 19:26 | |
| *** koolhead17 has quit IRC | 19:26 | |
| *** koolhead17 has joined #openstack-infra | 19:26 | |
| *** jerryz has quit IRC | 19:26 | |
| anteaya | awesome | 19:27 |
| anteaya | we neutron is networking so starting with that video doesn't hurt | 19:27 |
| anteaya | fungi can offer some guidance on security since he is on the vulnerability management team | 19:27 |
| sputnik13 | yeah I've heard a while back that neutron is going to be changing a lot | 19:27 |
| anteaya | and I don't know what you mean about management systems so I come up empty there | 19:28 |
| anteaya | sputnik13: neutron is constantly changing a lot | 19:28 |
| anteaya | like running through an earthquake | 19:28 |
| anteaya | but it is exciting! | 19:28 |
| sputnik13 | ceilometer would be related to management systems, being able to monitor and control | 19:28 |
| sputnik13 | and horizon for the management interfaces | 19:28 |
| openstackgerrit | Merged openstack/requirements: Add Werkzeug>=0.7 to global-requirements.txt https://review.openstack.org/134515 | 19:28 |
| sputnik13 | don't want to chew off too much at once though ;-) | 19:29 |
| anteaya | sputnik13: ah | 19:29 |
| anteaya | yeah, well you could do two things | 19:29 |
| anteaya | you could focus on one | 19:29 |
| anteaya | or you could be a generalist for two | 19:29 |
| anteaya | for instance neutron/horizon | 19:29 |
| anteaya | I don't think anyone is doing that work | 19:29 |
| *** koolhead17 has quit IRC | 19:30 | |
| anteaya | so it is up to you when you decide whether you want to specialize or generalize | 19:30 |
| tchaypo | sputnik13: welcome | 19:30 |
| fungi | yep, i can talk security thinges after meetings are wrapped up for the day | 19:30 |
| sputnik13 | tchaypo: thank you | 19:31 |
| *** emagana has quit IRC | 19:32 | |
| anteaya | sputnik13: I'm a generalist and work with infra as home base and also neutron and cinder as well as third party | 19:32 |
| *** emagana has joined #openstack-infra | 19:32 | |
| *** talluri has joined #openstack-infra | 19:32 | |
| *** emagana has quit IRC | 19:34 | |
| *** links has quit IRC | 19:34 | |
| *** emagana has joined #openstack-infra | 19:34 | |
| *** achanda has quit IRC | 19:34 | |
| *** emagana has quit IRC | 19:36 | |
| *** harlowja_away is now known as harlowja | 19:36 | |
| *** emagana has joined #openstack-infra | 19:36 | |
| sputnik13 | anteaya: cool, maybe one day I can cover multiple things too :-) | 19:37 |
| anteaya | find your comfort zone | 19:37 |
| sputnik13 | but I have to learn to walk before I can go running marathons ;-) | 19:37 |
| anteaya | get started somewhere, just don't lock yourself in early | 19:37 |
| anteaya | make sure you have room to move around | 19:37 |
| anteaya | yep | 19:37 |
| sputnik13 | anteaya: I hear you, thank you for the advice | 19:37 |
| anteaya | np | 19:37 |
| anteaya | happy to help | 19:37 |
| anteaya | let me know if I can help you find information on things | 19:38 |
| tchaypo | There's a guide someone wrote | 19:38 |
| anteaya | do share | 19:38 |
| tchaypo | gothicmindfood: is it public yet? | 19:39 |
| tchaypo | Is it even intended to public | 19:39 |
| *** yfried is now known as yfried|afk | 19:40 | |
| *** kmartin has joined #openstack-infra | 19:40 | |
| *** pelix has quit IRC | 19:42 | |
| clarkb | sdague: do you know where the e-r logs go? | 19:42 |
| clarkb | sdague: /var/log/elastic-recheck seems to be for the bot | 19:42 |
| clarkb | my next step is to try running by hand | 19:42 |
| sdague | hmmmm... it runs under cron | 19:42 |
| *** reed has quit IRC | 19:42 | |
| *** achanda has joined #openstack-infra | 19:43 | |
| sdague | maybe fungi knows? | 19:43 |
| *** dtalton has quit IRC | 19:43 | |
| sdague | running by hand is probably going to wedge on the lock | 19:43 |
| sdague | which is what's probably happened | 19:43 |
| sdague | the real issue is why the oldest job stalled | 19:43 |
| mtreinish | sdague: what lock? | 19:44 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Enable Ironic periodic-stable jobs https://review.openstack.org/135133 | 19:44 |
| clarkb | sdague: well there isn't a stalled job running | 19:44 |
| sdague | clarkb: oh... really? | 19:44 |
| clarkb | KeyError: '1282795' | 19:44 |
| sdague | that was the issue the last time | 19:44 |
| clarkb | http://paste.openstack.org/show/134492/ | 19:45 |
| fungi | sdague: clarkb: /var/spool/mail? | 19:46 |
| clarkb | fungi: its ok I think I figured it out | 19:46 |
| clarkb | and ya now that I look at it we aren't dev nulling that output so I should have it in mail somewhere | 19:46 |
| sdague | clarkb: hmph, lp api change? | 19:46 |
| clarkb | sdague: no someone made that bug private | 19:47 |
| sdague | oh, really? | 19:47 |
| clarkb | whyfor I have no idea | 19:47 |
| clarkb | fungi post meeting can you look at that bug and tell me if I can make it public? I see no reason not to | 19:47 |
| sdague | any idea who made it private? | 19:47 |
| sdague | ok, so that's a whole new thing that wasn't anticipated | 19:48 |
| mtreinish | clarkb, sdague: yeah I'm not sure why it's private: this bug was a duplicate of it: https://bugs.launchpad.net/openstack-ci/+bug/1331938 | 19:48 |
| clarkb | sdague: T C Okonski | 19:48 |
| *** yfried|afk is now known as yfried | 19:48 | |
| mtreinish | sdague: heh, yeah I don't think we ever considered that a gate bug would be marked private | 19:49 |
| clarkb | maybe catch that keyerror and print a graph that says lolfail? | 19:49 |
| clarkb | I don't want to print an empty graph as we may think we fixed the bug | 19:49 |
| clarkb | but some other indication it derped while not completely failing would be good | 19:49 |
| mtreinish | clarkb: we could just skip it and not print anything | 19:50 |
| fungi | clarkb: sure, though i need to sort of pay more attention than usual in tc and project meetings | 19:52 |
| *** sputnik13 has quit IRC | 19:52 | |
| clarkb | fungi: it should be short, its one you are familiar with | 19:52 |
| clarkb | I just wanted a quick check to ensure I wasn't missing something | 19:53 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Dynamically refresh events on task changes https://review.openstack.org/126542 | 19:53 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: handle possible private bugs in the graph https://review.openstack.org/135404 | 19:54 |
| gothicmindfood | tchaypo: was actually just going to circulate a final version to the team | 19:54 |
| sdague | clarkb / mtreinish ^^^ | 19:54 |
| gothicmindfood | tchaypo: I've spoken with mordred about what we might have to do to officially opensource/publish upstream | 19:54 |
| gothicmindfood | tchaypo: but I figure it's a document that'll make rounds no matter what :) | 19:55 |
| *** doug-fish1 has joined #openstack-infra | 19:55 | |
| *** rushiagr_away is now known as rushiagr | 19:55 | |
| * gothicmindfood will send out proposed-final version once her movers are gone today | 19:55 | |
| *** Guest67947 has joined #openstack-infra | 19:55 | |
| clarkb | gothicmindfood: oh are you joining us in portland land shortly? | 19:56 |
| fungi | clarkb: can do | 19:56 |
| *** doug-fish has quit IRC | 19:57 | |
| nibalizer | clarkb: https://review.openstack.org/#/c/126086/ is the review, and the failing test is the issue | 19:57 |
| nibalizer | so what i'd like to do is re run the tests and have you hold the node then go in as jenkins and figure out why jenkins can't ssh root@localhost | 19:58 |
| clarkb | nibalizer: rgr | 19:58 |
| *** sputnik13 has joined #openstack-infra | 19:58 | |
| nibalizer | got time now? | 19:58 |
| clarkb | nibalizer: uh not really | 19:58 |
| clarkb | and you probably want to sleep | 19:59 |
| nibalizer | its actaully only 8pm here, we can do it a different time tho | 19:59 |
| gothicmindfood | clarkb: yup - as of tomorrow, though I expect I will be air mattressing and cafe-interneting for a week or so before my things actually arrive/internet gets put on. | 19:59 |
| sdague | clarkb: so we should probably educate people about marking bugs private... because that seems like it might break more than er | 19:59 |
| *** yfried is now known as yfried|afk | 20:00 | |
| clarkb | sdague: yes its kind of a mean thing to do imo | 20:00 |
| clarkb | particularly if you are tracking a bug because it affects ou | 20:00 |
| sdague | yeh | 20:00 |
| sdague | is there justification for it? | 20:00 |
| clarkb | sdague: in this case best I can tell is no | 20:00 |
| sdague | or did someone do a thing and not realize it was bad | 20:00 |
| clarkb | going to have fungi double check | 20:00 |
| sdague | I'll chalk it up to a mistake | 20:00 |
| *** VijayTripathi has joined #openstack-infra | 20:00 | |
| sdague | mistakes happen | 20:00 |
| fungi | clarkb: got a bug number? | 20:01 |
| clarkb | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1282795 | 20:01 |
| fungi | sdague: yeah, n00b bug reporters accidentally switch bugs to private or private security all the time. i see a couple a week | 20:01 |
| anteaya | gothicmindfood: safe travels | 20:01 |
| *** teran has quit IRC | 20:01 | |
| clarkb | gothicmindfood: nice just in time for the rain to come back :) | 20:02 |
| fungi | clarkb: yep, that looks like it was a total drive-by accident/screwup | 20:02 |
| fungi | i've switched it back to public now | 20:02 |
| clarkb | cool | 20:02 |
| *** Ark has joined #openstack-infra | 20:03 | |
| *** lttrl has quit IRC | 20:03 | |
| *** Ark is now known as Guest22552 | 20:03 | |
| *** Sukhdev has joined #openstack-infra | 20:03 | |
| clarkb | ok now to do all those things I just said I would do | 20:04 |
| *** unicell has joined #openstack-infra | 20:04 | |
| *** achanda has quit IRC | 20:04 | |
| clarkb | asselin: nibalizer fwiw https://review.openstack.org/#/c/134373/ re that change I think I am ok with just putting that into the imported repo as the most recent commit. Not sure if that makes more work for you guys though | 20:05 |
| clarkb | I am also ok with reviewing and merging those changes via gerrit after the import :) | 20:05 |
| *** cpowell has quit IRC | 20:06 | |
| nibalizer | i like gerrit review after import | 20:07 |
| gothicmindfood | anteaya: thanks! | 20:07 |
| *** cikid has joined #openstack-infra | 20:07 | |
| *** ociuhandu has quit IRC | 20:07 | |
| *** talluri has quit IRC | 20:09 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Clarify changes needed for new puppet repo https://review.openstack.org/134393 | 20:10 |
| *** aysyd has quit IRC | 20:11 | |
| clarkb | is bashate now voting on system-config? | 20:14 |
| *** Guest67947 has quit IRC | 20:14 | |
| clarkb | nibalizer: your source modules env change fails bashate | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| *** ZZelle has joined #openstack-infra | 20:15 | |
| nibalizer | clarkb: oh okay ill fix that | 20:15 |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Add required puppet module files and .gitreview https://review.openstack.org/134373 | 20:15 |
| *** amuller__ is now known as amuller | 20:16 | |
| ianw | clarkb / fungi / sdague : do you think there might be a case for adding more people to +2 to devstack-gate? it seems lots have one +2 from an interested/involved person, but are waiting for someone else to sanity check that second +2 | 20:16 |
| nibalizer | clarkb: fixed | 20:16 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Source modules.env from apply-test.sh https://review.openstack.org/134723 | 20:17 |
| clarkb | nibalizer: thank you, will review now | 20:17 |
| sdague | ianw: people are just filtering back from paris | 20:17 |
| clarkb | nibalizer: why is declare -A MODULES commented out? | 20:19 |
| *** jesusaurus has quit IRC | 20:19 | |
| nibalizer | oh i think we can remove that | 20:19 |
| *** yfried|afk is now known as yfried | 20:20 | |
| anteaya | clarkb: yes bashate is now voting on system-config | 20:20 |
| clarkb | nibalizer: hold on another suggestion incoming | 20:21 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Source modules.env from apply-test.sh https://review.openstack.org/134723 | 20:22 |
| *** wendar_ is now known as wendar | 20:22 | |
| nibalizer | clarkb: erp, okay | 20:22 |
| nibalizer | i think i just made it better anyways | 20:22 |
| nibalizer | bash requires those declare statements, its kinda weird that we put the declare statements in install_modules.sh instead of modules.env but /me doesn't really care | 20:22 |
| nibalizer | now its consistent :P | 20:22 |
| clarkb | nibalizer: sort of, I don't think apply-test will work in that case | 20:22 |
| *** alexpilotti has quit IRC | 20:23 | |
| clarkb | because you onl declare the array in the other location | 20:23 |
| clarkb | oh you do it in both... | 20:23 |
| clarkb | that just adds different duplication | 20:23 |
| nibalizer | so i just pushed a ps that has declare -A in both install_modules.sh and apply-test.sh | 20:23 |
| nibalizer | im happy to change it | 20:23 |
| clarkb | nibalizer: that should be fine as an intermediate step | 20:23 |
| clarkb | see comment on ps4 | 20:23 |
| nibalizer | okay | 20:23 |
| *** alexpilotti has joined #openstack-infra | 20:24 | |
| *** alexpilotti has quit IRC | 20:25 | |
| nibalizer | clarkb: ah so what is the best way to describe openstack-infra/puppet-jenkins ? | 20:25 |
| clarkb | nibalizer: thats the long name | 20:26 |
| nibalizer | the reason its 'short' is because its not a git uri any more | 20:26 |
| nibalizer | okay so project_long_names would be acceptable to you? | 20:26 |
| *** sarob has quit IRC | 20:26 | |
| clarkb | so typically we have code all over that says short_name = basname longname | 20:26 |
| stevebaker | anteaya: Enabling the dib service just clones some git repos. The change which builds the custom image is https://review.openstack.org/#/c/113711/ which should only land if it proves stable | 20:26 |
| clarkb | nibalizer: or just project_names | 20:26 |
| nibalizer | okay ill do project_names | 20:26 |
| *** sputnik13 has quit IRC | 20:26 | |
| *** chandankumar has quit IRC | 20:27 | |
| clarkb | sounds good | 20:27 |
| *** zul has joined #openstack-infra | 20:27 | |
| asselin | clarkb, ok. I submit a follow on patch to address remaining issues. I do agree it's good to have that as first commit post split. | 20:28 |
| anteaya | stevebaker: can I get that information in the commit message? | 20:29 |
| *** ildikov has joined #openstack-infra | 20:29 | |
| stevebaker | anteaya: sure thing | 20:30 |
| clarkb | oh krotscheck based the git review doc update on another unmerged changed that explains why it isn't going in the gate | 20:31 |
| * clarkb reivews the other change | 20:31 | |
| anteaya | stevebaker: thanks | 20:31 |
| *** andreykurilin_ has joined #openstack-infra | 20:32 | |
| clarkb | fungi: FYI I approved that other change without a second +2. it was just a manpage update and man -l confirmed the edits do the right thing | 20:32 |
| nibalizer | clarkb: see my response on ps4 | 20:33 |
| clarkb | nibalizer: so I didn't realize we were flattening URLs there | 20:34 |
| clarkb | I thought they were just listed as project names for some reason | 20:34 |
| openstackgerrit | Steve Baker proposed openstack-infra/project-config: Enable dib service for heat-dsvm-functional https://review.openstack.org/134422 | 20:34 |
| *** mfink__ has joined #openstack-infra | 20:34 | |
| clarkb | which made that loop a noop | 20:34 |
| clarkb | but it isn't a noop so ya if you just rename the thing I think that clears up the confusion in my head | 20:34 |
| nibalizer | ah yea, its kindof data manipulation in bash | 20:34 |
| nibalizer | sweet | 20:34 |
| fungi | clarkb: cool yo | 20:35 |
| nibalizer | ok ps6 inc | 20:35 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Source modules.env from apply-test.sh https://review.openstack.org/134723 | 20:35 |
| *** mfink has quit IRC | 20:35 | |
| clarkb | now we just need zuul to get through 4k events and progress can be made on new things :) | 20:36 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Allow IPA to be cloned and built https://review.openstack.org/134438 | 20:36 |
| *** yjiang5 is now known as yjiang5_lunch | 20:36 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/devstack-gate: Update usage of LOCALRC_OLDNEW https://review.openstack.org/135418 | 20:37 |
| *** alexpilotti has joined #openstack-infra | 20:38 | |
| *** yfried is now known as yfried|afk | 20:38 | |
| *** alexpilotti has quit IRC | 20:39 | |
| *** amuller has quit IRC | 20:39 | |
| *** sarob has joined #openstack-infra | 20:39 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Puppet Jenkins uses storyboard https://review.openstack.org/135420 | 20:40 |
| *** alexpilotti has joined #openstack-infra | 20:40 | |
| *** alexpilotti has quit IRC | 20:41 | |
| *** andreykurilin_ has quit IRC | 20:41 | |
| *** mattfarina has joined #openstack-infra | 20:42 | |
| *** andreykurilin_ has joined #openstack-infra | 20:42 | |
| *** yfried|afk is now known as yfried | 20:42 | |
| *** yamamoto has joined #openstack-infra | 20:42 | |
| *** alexpilotti has joined #openstack-infra | 20:42 | |
| *** alexpilotti has quit IRC | 20:43 | |
| *** bknudson has quit IRC | 20:44 | |
| *** alexpilotti has joined #openstack-infra | 20:45 | |
| *** alexpilotti has quit IRC | 20:46 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: handle possible private bugs in the graph https://review.openstack.org/135404 | 20:46 |
| clarkb | fungi: ok voted on that stackforge third party change and responded to the list | 20:47 |
| *** alexpilotti has joined #openstack-infra | 20:47 | |
| *** yamamoto has quit IRC | 20:47 | |
| clarkb | git review storyboard update should go in shortly | 20:47 |
| openstackgerrit | Nels proposed openstack-infra/devstack-gate: Adding support for DEVSTACK_GATE_LIBVIRT_TYPE https://review.openstack.org/134348 | 20:48 |
| nibalizer | fungi: do you know when you will force push to puppet-httpd ? | 20:48 |
| *** alexpilotti has quit IRC | 20:48 | |
| *** alexpilotti has joined #openstack-infra | 20:48 | |
| *** alexpilotti has quit IRC | 20:49 | |
| fungi | nibalizer: i'm trying not to get too distracted and miss the tc and project meeting discussions, but will shift gears after they're over | 20:49 |
| clarkb | e-r pages are updated now | 20:49 |
| nibalizer | ah oaky, yea go do meetings, important stuff | 20:49 |
| *** alexpilotti has joined #openstack-infra | 20:51 | |
| mtreinish | nibalizer: oh is refreshonly all that's missing for the db-manage exec to make it work like I want now? | 20:51 |
| *** alexpilotti has quit IRC | 20:51 | |
| *** Guest22552 has quit IRC | 20:51 | |
| openstackgerrit | Merged openstack-infra/git-review: Fix ---list and "departement" typos in man page https://review.openstack.org/127256 | 20:52 |
| openstackgerrit | Merged openstack-infra/git-review: Switched documentation to point to storyboard https://review.openstack.org/133847 | 20:52 |
| clarkb | boom, just waiting on pypi-mirror now | 20:52 |
| *** todd_dsm has quit IRC | 20:52 | |
| clarkb | krotscheck: fungi ^ | 20:52 |
| mtreinish | nibalizer: if so clarkb misled me :) | 20:52 |
| nibalizer | mtreinish: ya that was the thing i reviewed this morning or something right? | 20:52 |
| *** otherwiseguy has joined #openstack-infra | 20:53 | |
| *** alexpilotti has joined #openstack-infra | 20:53 | |
| clarkb | mtreinish: nibalizer pip provider is bad and does the wrong thing with latest | 20:53 |
| clarkb | it will fire each and every time | 20:53 |
| dstufft | some might even said pip itself is b ad | 20:53 |
| dstufft | :/ | 20:53 |
| dstufft | say* | 20:53 |
| clarkb | dstufft: if it helps the pip puppet provider is worse | 20:53 |
| clarkb | I guess it can't be better | 20:53 |
| clarkb | so it has that working aginast it | 20:54 |
| *** alexpilotti has quit IRC | 20:54 | |
| dstufft | clarkb: pip - at least we're not the worst | 20:54 |
| fungi | clarkb: oh, and https://review.openstack.org/133863 has been running in production since last week and openstackgerrit is still around without our assistance. i still need to check the logs for signs that code path was exercised though | 20:54 |
| *** jesusaurus has joined #openstack-infra | 20:55 | |
| *** alexpilotti has joined #openstack-infra | 20:55 | |
| mtreinish | clarkb: oh well, so setting refreshonly won't work then. I'm just trying to figure out how to respin that patch | 20:56 |
| *** alexpilotti has quit IRC | 20:56 | |
| clarkb | fungi: cool will take a look | 20:56 |
| clarkb | mtreinish: did we decide not to install from source? | 20:56 |
| * clarkb feels like he is missing something | 20:56 | |
| *** alexpilotti has joined #openstack-infra | 20:57 | |
| mtreinish | clarkb: no I'm just trying to avoid installing from source because as basically the sole contributor and reviewer I really didn't want to make CD guarantees on the repo | 20:57 |
| mtreinish | and I'm trying to figure out if I have to use https://review.openstack.org/135161 and just manually update tags on each release or if there is a better way | 20:58 |
| *** annegent_ has joined #openstack-infra | 20:58 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Setup ssh_known_host files for aiopcpu https://review.openstack.org/134670 | 20:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests https://review.openstack.org/131604 | 20:58 |
| *** AlexF has joined #openstack-infra | 20:58 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 20:58 |
| *** alexpilotti has quit IRC | 20:59 | |
| sdague | clarkb: er working again | 20:59 |
| sdague | thanks | 20:59 |
| clarkb | sdague: yup np | 20:59 |
| clarkb | you and fungi did the actual fixing | 20:59 |
| sdague | clarkb: yeh, but I'd never been able to know what broke without you | 21:00 |
| clarkb | mtreinish: it is a bit annoying I think you can manuall bump that tag version for now | 21:01 |
| clarkb | mtreinish: or pip install a specifric version not from source but from pypi | 21:01 |
| openstackgerrit | Nels proposed openstack-infra/devstack-gate: Adding support for DEVSTACK_GATE_LIBVIRT_TYPE https://review.openstack.org/134348 | 21:01 |
| clarkb | mtreinish: that should work. iirc the problem is when you tell pip provider to ensure latest | 21:01 |
| *** otherwiseguy has quit IRC | 21:01 | |
| nibalizer | clarkb: i did some testing on my laptop and had no porblem | 21:02 |
| clarkb | nibalizer: maybe newer puppet fixed it? | 21:02 |
| clarkb | we are on latest puppet 3 now | 21:02 |
| *** otherwiseguy has joined #openstack-infra | 21:03 | |
| clarkb | in that case maybe give it a go? | 21:03 |
| clarkb | mtreinish: ^ | 21:03 |
| *** jd__ has quit IRC | 21:03 | |
| mtreinish | ok cool, I'll respin that patch to just use refreshonly | 21:03 |
| nibalizer | http://paste.ubuntu.com/9078478/ | 21:03 |
| anteaya | ttx the patch(es) for updating plt info in governance, one big patch or separate, independant changes? | 21:03 |
| nibalizer | you can play with that snippet and get it to do the stuff | 21:03 |
| *** bknudson has joined #openstack-infra | 21:04 | |
| *** achanda has joined #openstack-infra | 21:05 | |
| *** annegent_ has quit IRC | 21:05 | |
| tchaypo | So getting back to psr | 21:06 |
| *** otter768 has quit IRC | 21:06 | |
| tchaypo | There's going to need to be a wrapper around zuul/merger/merger.cmd | 21:07 |
| tchaypo | I could push psr as a new tool which depends on zuul | 21:07 |
| *** jd__ has joined #openstack-infra | 21:07 | |
| tchaypo | Or I could relocate the cli I to zuul and put it there | 21:08 |
| tchaypo | Any thoughts? | 21:08 |
| *** baoli has joined #openstack-infra | 21:08 | |
| *** atiwari has quit IRC | 21:08 | |
| clarkb | I am not sure why it needs to depend on the zuul merger | 21:08 |
| *** VijayTripathi has quit IRC | 21:08 | |
| clarkb | couldn;t it just apply a series of refs locally in the cloner tool? | 21:08 |
| *** baoli has quit IRC | 21:09 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove python-heatclient from integrated gate https://review.openstack.org/135426 | 21:09 |
| *** achanda_ has joined #openstack-infra | 21:09 | |
| *** achanda has quit IRC | 21:09 | |
| *** hdd has quit IRC | 21:10 | |
| *** baoli has joined #openstack-infra | 21:10 | |
| sdague | anteaya / clarkb ^^^ I missed one in the libs from pypi conversion | 21:10 |
| clarkb | +2 :) | 21:10 |
| anteaya | I was just looking at that | 21:10 |
| *** teran has joined #openstack-infra | 21:10 | |
| anteaya | sdague: have you a link to the original patch that missed this one? | 21:11 |
| *** SumitNaiksatam has quit IRC | 21:11 | |
| *** mwagner_lap has quit IRC | 21:11 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add refreshonly to 'upgrade_subunit2sql_db' https://review.openstack.org/135161 | 21:12 |
| mtreinish | clarkb, nibalizer: ^^^ I think that should do | 21:13 |
| sdague | anteaya: one sec | 21:13 |
| anteaya | thanks | 21:13 |
| mtreinish | let me know if I screwed up the commit msg too much | 21:13 |
| mordred | tchaypo: I'd vote for cli in zuul | 21:14 |
| sdague | anteaya: https://review.openstack.org/#/c/132546/10/zuul/layout.yaml,cm | 21:14 |
| anteaya | thanks | 21:14 |
| *** dustins has quit IRC | 21:15 | |
| *** bknudson has quit IRC | 21:15 | |
| anteaya | the view looks different | 21:15 |
| *** yfried has quit IRC | 21:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add refreshonly to 'upgrade_subunit2sql_db' https://review.openstack.org/135161 | 21:16 |
| clarkb | mtreinish: +2 | 21:16 |
| mtreinish | clarkb: cool, thanks | 21:18 |
| sdague | anteaya: that's new screen | 21:18 |
| sdague | I started using new screen at summit, after a few adjustments, I do like it better (mostly for the patch series bit) | 21:19 |
| clarkb | sdague: what patchseries bit? | 21:19 |
| nibalizer | mtreinish: +1 | 21:19 |
| clarkb | mtreinish: we can yell at fungi to merge that when meetings end | 21:19 |
| clarkb | nibalizer: ok you going to be awake for the next hour or so? | 21:20 |
| nibalizer | clarkb: ya | 21:20 |
| clarkb | nibalizer: I think I ma have cleared enough backlog to debug that thing | 21:20 |
| nibalizer | oh sweet | 21:20 |
| fungi | clarkb: mtreinish: sure, yelling soon is fine | 21:20 |
| mtreinish | clarkb: ok sure, I also need to poke him about subunit-worker01 | 21:20 |
| JayF | Do devstack-gate changes take effect immediately, or do i need to wait some time for nodes to cycle? | 21:20 |
| clarkb | JayF: immediately | 21:20 |
| JayF | sweet | 21:21 |
| * JayF rechecks an IPA change | 21:21 | |
| clarkb | nibalizer: can has link again? | 21:21 |
| anteaya | sdague: python-ganttclient? pylockfile? | 21:21 |
| openstackgerrit | Merged openstack-infra/pypi-mirror: Switched documentation to point to storyboard https://review.openstack.org/133859 | 21:21 |
| clarkb | krotscheck: fungi ^ the last of the doc updates. now we need to lock down projects | 21:21 |
| sdague | clarkb: the "related changes" | 21:21 |
| nibalizer | clarkb: https://review.openstack.org/#/c/126086/ | 21:21 |
| anteaya | sdague: they still have the integrated-gate job macro, should they be removed too? | 21:21 |
| clarkb | sdague: gah you say "patch" when you mean "change" :P and I always get confused | 21:22 |
| sdague | anteaya: eventually | 21:22 |
| clarkb | sdague: I really dislike how c2 treats patchsets | 21:22 |
| *** bknudson has joined #openstack-infra | 21:22 | |
| sdague | clarkb: heh | 21:22 |
| clarkb | (it hides them all) | 21:22 |
| sdague | c2? | 21:22 |
| clarkb | change screen two | 21:22 |
| anteaya | sdague: care to make a note of that in the commit message? | 21:22 |
| clarkb | c2 is the url prefix | 21:22 |
| *** bknudson1 has joined #openstack-infra | 21:22 | |
| sdague | anteaya: I think it's redundant, this has been successive waves | 21:22 |
| anteaya | I didn't know that | 21:23 |
| anteaya | I will make notes then | 21:23 |
| anteaya | fungi approved my input isn't necessary | 21:23 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Source modules.env from apply-test.sh https://review.openstack.org/134723 | 21:23 |
| clarkb | nibalizer: ok I rechecked that change now waiting for test to start. | 21:24 |
| * clarkb watching it like a hawk | 21:24 | |
| *** sputnik13 has joined #openstack-infra | 21:25 | |
| nibalizer | clarkb: tyty | 21:25 |
| nibalizer | so the jenkins user should be able to ssh root@localhost but for some reason it can | 21:25 |
| nibalizer | t | 21:26 |
| nibalizer | likely it will be pretty easy for you to figure it out on the box (or thats what im hoping) | 21:26 |
| *** bknudson has quit IRC | 21:26 | |
| *** stevemar has quit IRC | 21:26 | |
| clarkb | ok will test once we have the node | 21:26 |
| *** hdd has joined #openstack-infra | 21:28 | |
| krotscheck | clarkb, fungi: I can’t lock down projects, but I’ll go get the email ready. | 21:30 |
| clarkb | nibalizer: caught the node will let the test finish before I dig around | 21:31 |
| clarkb | to avoid any interference | 21:31 |
| *** AlexF has quit IRC | 21:33 | |
| *** mfink has joined #openstack-infra | 21:34 | |
| *** yjiang5_lunch is now known as yjiang5 | 21:34 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/zuul: Add gerrit owner information to status.json https://review.openstack.org/135386 | 21:36 |
| *** mfink__ has quit IRC | 21:36 | |
| *** hdd has quit IRC | 21:37 | |
| *** rlandy has quit IRC | 21:38 | |
| *** mmaglana has quit IRC | 21:39 | |
| nibalizer | okay | 21:39 |
| *** lttrl has joined #openstack-infra | 21:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Ignore playground directories for translation https://review.openstack.org/133368 | 21:40 |
| *** miqui has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack-infra/project-config: remove python-heatclient from integrated gate https://review.openstack.org/135426 | 21:44 |
| *** baoli has quit IRC | 21:45 | |
| *** baoli has joined #openstack-infra | 21:46 | |
| lifeless | headsup: just released suubunit 1.0.0 | 21:46 |
| *** erikwilson has quit IRC | 21:46 | |
| lifeless | not expecting any fallout, but who knows :) | 21:46 |
| *** gokrokve_ has joined #openstack-infra | 21:46 | |
| mtreinish | lifeless: heh :) | 21:47 |
| clarkb | thanks for the heads up | 21:47 |
| nibalizer | clarkb: it should be done by now i would expect | 21:47 |
| mtreinish | lifeless: does that include the changes to handle the discovery failure tracebacks? | 21:47 |
| clarkb | nibalizer: ya doing all sorts of context switching let me look | 21:48 |
| *** baoli has quit IRC | 21:48 | |
| *** rushiagr is now known as rushiagr_away | 21:48 | |
| clarkb | nibalizer: permission denied public key | 21:48 |
| clarkb | now to debug | 21:48 |
| *** baoli has joined #openstack-infra | 21:48 | |
| lifeless | mtreinish: yes | 21:48 |
| mtreinish | lifeless: awesome | 21:49 |
| *** SumitNaiksatam has joined #openstack-infra | 21:50 | |
| *** gokrokve has quit IRC | 21:50 | |
| *** lttrl has quit IRC | 21:50 | |
| clarkb | hrm ssh debug output doesn't help much. its using id_rsa then giving up and going to the other keys (that don't exist) but I don't see why it is giving up on id_rsa | 21:50 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/zuul: EXPERIMENT: to see if jobs run properly https://review.openstack.org/135434 | 21:51 |
| mtreinish | clarkb: are you using -vvv ? | 21:51 |
| nibalizer | i maybe its the 'localhost' lines we put in sshd_config ? | 21:51 |
| mtreinish | although even that is rarely helpful :) | 21:51 |
| nibalizer | maybe localhost isn't being recognized? | 21:51 |
| lifeless | mtreinish: as long as testtools 1.4.0 is in used | 21:52 |
| lifeless | erm | 21:52 |
| lifeless | in use | 21:52 |
| lifeless | actually, I might be wrong. It might need 1.5.0. Shrug :) | 21:52 |
| clarkb | mtreinish: ya | 21:52 |
| *** mbacchi has quit IRC | 21:53 | |
| mtreinish | lifeless: heh, ok I'll push to get some reviews on https://review.openstack.org/135115 because either way we'll need it to get around the attr thing with tempest | 21:53 |
| clarkb | nibalizer: so I just put jenkins key in my authorized keys and that worked. so I think this may be related to root login without password stuff | 21:53 |
| fungi | lifeless: also we blocked 1.4.0 for other reasons anyway | 21:53 |
| fungi | lifeless: so we'd presumably need 1.5.0 regardless | 21:53 |
| *** baoli has quit IRC | 21:53 | |
| *** mpaolino has joined #openstack-infra | 21:54 | |
| *** mpaolino has quit IRC | 21:54 | |
| fungi | krotscheck: clarkb: yeah i need to rerun the imports one more time before i lock them down anyway | 21:54 |
| lifeless | fungi: no :) | 21:54 |
| mtreinish | fungi: we can work around the block with that ^^^ patch I think | 21:54 |
| lifeless | fungi: 1.4.0 is blocked until tempest-lib unbreaks its buggy MRO order | 21:54 |
| clarkb | nibalizer: I am hacking the sshd_config | 21:54 |
| fungi | mtreinish: oh, are we going to be able to lift the !=1.4.0 now? | 21:54 |
| fungi | awesome | 21:54 |
| *** baoli has joined #openstack-infra | 21:54 | |
| mtreinish | fungi: well when 135115 lands, it at least fixed it for me in quick local testing | 21:54 |
| lifeless | yeah, its 1.5.0 that will have the full support | 21:55 |
| lifeless | nuts, bad changelog in subunit. ah well. | 21:56 |
| *** banix has quit IRC | 21:56 | |
| *** mfink has quit IRC | 21:56 | |
| *** Kiall has quit IRC | 21:56 | |
| nibalizer | clarkb: okay that lines up with my thinking | 21:57 |
| *** mattfarina has quit IRC | 21:58 | |
| lifeless | mtreinish: this is what master testtools gets me: | 21:58 |
| lifeless | http://paste.openstack.org/show/134500/ | 21:58 |
| mtreinish | lifeless: ooh, very nice | 21:59 |
| mtreinish | I think that's exactly what everyone is looking for :) | 21:59 |
| lifeless | the yak shave on this was insane | 21:59 |
| clarkb | nibalizer: ok just straight up allowing root login didn't help | 21:59 |
| clarkb | so something else is going on | 22:00 |
| clarkb | is this maybe one of those things where ubuntu is like lol no root for you? | 22:00 |
| lifeless | mtreinish: so later today 1.5.0 is going to come out I hope | 22:00 |
| lifeless | mtreinish: you *might* want to get 135115 landed so it doesn't break things again :) | 22:00 |
| mtreinish | lifeless: heh, I was just going to ask if you can hold off until it lands :) | 22:00 |
| clarkb | pam.d/sshd looks fine | 22:01 |
| anteaya | ttx one giant governance patch? | 22:01 |
| anteaya | ttx or small separate ones? | 22:01 |
| lifeless | mtreinish: I have one more patch I want to try and get together, but inventory bad. | 22:01 |
| anteaya | then good-night | 22:01 |
| ttx | anteaya: giant works | 22:02 |
| anteaya | very good | 22:02 |
| ttx | that's what we did last time iirc | 22:02 |
| nibalizer | clarkb: hrm | 22:02 |
| clarkb | sshd[6019]: ROOT LOGIN REFUSED FROM 127.0.0.1 | 22:02 |
| anteaya | sounds good, good night | 22:02 |
| ttx | anteaya: an "election results" change | 22:03 |
| nibalizer | so all the PermitRootLogin lines are 'yes' ? | 22:03 |
| nibalizer | did you restart sshd after you changed the config? | 22:03 |
| clarkb | nibalizer: I removed the second one and set the other to yes | 22:03 |
| *** lttrl has joined #openstack-infra | 22:03 | |
| clarkb | and yes restarted ssh | 22:03 |
| *** bknudson1 has quit IRC | 22:03 | |
| nibalizer | :/ | 22:03 |
| *** Sukhdev has quit IRC | 22:03 | |
| nibalizer | maybe try sshing into its public ip address? | 22:03 |
| nibalizer | or if it doesnt have a public then a not localhost one? | 22:04 |
| anteaya | ttx I will call it such | 22:04 |
| *** bknudson has joined #openstack-infra | 22:05 | |
| *** emagana has quit IRC | 22:05 | |
| *** jaypipes has quit IRC | 22:05 | |
| *** bswartz has quit IRC | 22:05 | |
| jesusaurus | nibalizer: for the forge-upload job, how does the project get bootstrapped? is that a manual step of one of the rooters creating the project in the forge? | 22:05 |
| *** emagana has joined #openstack-infra | 22:06 | |
| nibalizer | well you see thats a good question | 22:06 |
| fungi | jesusaurus: if so, that's no different than the other distribution networks to which we upload releases | 22:06 |
| clarkb | nibalizer: nope | 22:06 |
| clarkb | this is weird | 22:06 |
| nibalizer | clarkb: yea very weird | 22:06 |
| *** changbl has quit IRC | 22:06 | |
| nibalizer | oh is the root user locked or something? | 22:07 |
| nibalizer | or /bin/nologin as shell or something? | 22:07 |
| jesusaurus | fungi: yeah, just something to document | 22:07 |
| fungi | mtreinish: i think i'm going to have to blow away the subunit-worker01 instance and try launching it again | 22:07 |
| clarkb | nibalizer: its bash | 22:07 |
| nibalizer | jesusaurus: fungi ya i would say probably yes someone would have to make the modules manually on the website the first time | 22:07 |
| clarkb | and an x in /etc/passwd password | 22:08 |
| mtreinish | fungi: ok I'll be here waiting to see what else blows up :) | 22:08 |
| clarkb | and shadow looks normal | 22:08 |
| clarkb | I think it must be pam related | 22:08 |
| clarkb | and how ubuntu et al think you shouldn't root | 22:08 |
| nibalizer | ya | 22:08 |
| asselin | does this need any update to puppet split spec? I don't really understand it "AGREED: each puppet module needs to be associated with the infra project group (fungi, 19:15:26)" | 22:08 |
| nibalizer | does shadow have the '!' or the 'x' ? | 22:09 |
| clarkb | shadow has a hash | 22:10 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Initial magnum client import for stackforge https://review.openstack.org/135436 | 22:10 |
| *** emagana has quit IRC | 22:10 | |
| dims | haha, zuul's jobs are busted - https://review.openstack.org/#/c/135434/ | 22:10 |
| nibalizer | clarkb: huh | 22:11 |
| openstackgerrit | Merged openstack-infra/system-config: Add refreshonly to 'upgrade_subunit2sql_db' https://review.openstack.org/135161 | 22:12 |
| ianw | is there a mid-cycle infra meetup in any sort of planning stage? | 22:13 |
| fungi | asselin: referring to the groups list in the projects.yaml | 22:13 |
| nibalizer | clarkb: whant to try setting UesPAM no in sshd_config ? | 22:13 |
| mtreinish | ianw: jeblair and I were talking about it at summit and I think the decision was to not have an infra or qa midcycle this time | 22:13 |
| fungi | ianw: none so far as i know. we usually don't have one unless there's a specific task we're trying to tackle (an onboarding effort, a topic-relevant sprint, something we want to work on) | 22:14 |
| asselin | fungi, currently it's groups: | 22:14 |
| asselin | - openstack-ci | 22:14 |
| fungi | asselin: yep, that's what i meant | 22:14 |
| fungi | asselin: i was being vague since once we finish switching over to storyboard we're going to rename that group to infra | 22:15 |
| fungi | asselin: but at the moment, in an effort to support our automated bug updating, we're keeping it the same as the blanket project name we're using in launchpad | 22:15 |
| clarkb | nibalizer: gah! | 22:15 |
| clarkb | nibalizer: so I think the ssh init script on ubuntu is broken | 22:15 |
| ianw | mtreinish / fungi : thanks | 22:16 |
| clarkb | nibalizer: if you use `restart ssh` that seems to have worked | 22:16 |
| clarkb | nibalizer: what does your job do? | 22:16 |
| nibalizer | clarkb: restart i think | 22:16 |
| asselin | fungi, ok, I see. So no updates now. Updates coming later. | 22:16 |
| fungi | asselin: right | 22:16 |
| nibalizer | ya its a restart | 22:16 |
| nibalizer | http://logs.openstack.org/86/126086/3/check/gate-puppet-storyboard-puppet-beaker-rspec/2394815/console.html#_2014-11-18_17_30_30_669 | 22:16 |
| clarkb | ok I restored your sshd_config and tried and that doesn't work. so I made changes that did fix it in between somewhere | 22:17 |
| clarkb | but at least this is progress | 22:17 |
| nibalizer | yes | 22:17 |
| *** lttrl has quit IRC | 22:18 | |
| nibalizer | so we just need to get to an sshd_config that will work | 22:18 |
| clarkb | nibalizer: if I change the toplevel PermitRootLogin to without-password that works | 22:18 |
| clarkb | nibalizer: let me see if I can make your addition work with tweaks | 22:18 |
| nibalizer | okay, so if i add a sed -i to my test? | 22:18 |
| nibalizer | that would be okay? | 22:18 |
| clarkb | nibalizer: well let me see if I can make the other thing work first | 22:18 |
| nibalizer | good idea | 22:18 |
| *** bcourt has quit IRC | 22:20 | |
| clarkb | nibalizer: match address 127.0.0.1 works | 22:21 |
| clarkb | nibalizer: so I think host localhost isn't matched for whatever raisin | 22:22 |
| nibalizer | ya okay | 22:22 |
| *** bhuvan has quit IRC | 22:22 | |
| *** baoli has quit IRC | 22:22 | |
| *** otherwiseguy has quit IRC | 22:22 | |
| nibalizer | sweet so if i replace loalhost with 127.0.0.1 we're probably good? | 22:22 |
| clarkb | also address ::1 | 22:22 |
| nibalizer | okay | 22:22 |
| *** baoli has joined #openstack-infra | 22:22 | |
| mtreinish | clarkb: is it in /etc/hosts I've seen some weird things around 127.0.0.1, localhost, and /etc/hosts before | 22:22 |
| clarkb | ya I think so add those two and even ssh root@localhost works | 22:23 |
| anteaya | dhellmann: you were waiting for me :D | 22:23 |
| clarkb | I am going to delete that node now | 22:23 |
| dhellmann | anteaya: that change hit my inbox as I was catching up on email :-) | 22:23 |
| anteaya | timing | 22:24 |
| *** bhuvan has joined #openstack-infra | 22:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Fix up sudo wildcard copy https://review.openstack.org/133914 | 22:26 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use ip addresses for localhost in beaker tests https://review.openstack.org/135442 | 22:26 |
| nibalizer | clarkb: thanks for your help | 22:26 |
| clarkb | https://jenkins02.openstack.org/job/check-tempest-dsvm-aiopcpu/24/console I think there may be a nodepool bug with keeping subnodes around | 22:27 |
| nibalizer | that review ^^ is the changes we figured out if you want to review it while its hot in your mind | 22:27 |
| clarkb | nibalizer: match address not match host | 22:27 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/devstack-gate: Update usage of LOCALRC_OLDNEW https://review.openstack.org/135418 | 22:28 |
| clarkb | ianw: we may be able to stop caring about tempest nosetest stuff completely at this point | 22:28 |
| clarkb | mtreinish: ^ | 22:28 |
| clarkb | is icehouse new enough to not tempest nose? | 22:29 |
| nibalizer | clarkb: oh okay | 22:29 |
| *** sarob has quit IRC | 22:29 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use ip addresses for localhost in beaker tests https://review.openstack.org/135442 | 22:29 |
| *** shashankhegde has quit IRC | 22:29 | |
| mtreinish | clarkb: yeah, nose was deprecated in havana and unsupported in icehouse | 22:30 |
| *** otherwiseguy has joined #openstack-infra | 22:31 | |
| clarkb | ianw: ^ want to just rip out the nose stuff instead? | 22:31 |
| *** lttrl has joined #openstack-infra | 22:31 | |
| *** sputnik13 has quit IRC | 22:31 | |
| *** gokrokve_ has quit IRC | 22:31 | |
| mtreinish | although we still get tons of bug reports for people running with nose (despite the fact that a bunch of things don't work with it) | 22:31 |
| *** baoli has quit IRC | 22:32 | |
| *** esker has quit IRC | 22:32 | |
| clarkb | that shouldn't be an issue for d-g imo | 22:33 |
| *** baoli has joined #openstack-infra | 22:33 | |
| *** banix has joined #openstack-infra | 22:33 | |
| *** sarob has joined #openstack-infra | 22:33 | |
| *** mfink has joined #openstack-infra | 22:34 | |
| ianw | clarkb: there's a review out for that https://review.openstack.org/#/c/131714/ | 22:34 |
| adam_g | anyone have a sec for https://review.openstack.org/135418 ? that multi node dsg patch broke debug log collection that we need to debug a recurring ironic gate failure | 22:34 |
| ianw | clarkb: i can rebase on it, if you want | 22:34 |
| *** esker has joined #openstack-infra | 22:34 | |
| clarkb | ianw: ya why don't we do that instead | 22:34 |
| mtreinish | my favorite of which is that *test* is the method name match that nose uses for test discovery which picks up a bunch of helper methods in tempest :) | 22:34 |
| * clarkb reviews that other change too | 22:34 | |
| *** sputnik13 has joined #openstack-infra | 22:34 | |
| fungi | mtreinish: weird, still getting... Could not find dependency Class[Logstash::Indexer] for Service[jenkins-subunit-worker-B] at /opt/system-config/production/modules/subunit2sql/manifests/worker.pp:65 | 22:35 |
| fungi | mtreinish: i'll play around with this a little and see if i can solve it | 22:35 |
| ianw | clarkb: if you're likely to approve https://review.openstack.org/#/c/131714/ i won't waste time rebasing on it now, i'll just wait for it to hit master | 22:35 |
| mtreinish | fungi: hmm, yeah that's weird | 22:35 |
| clarkb | ianw: cool let me do that real quick so I can tell you :) | 22:35 |
| mtreinish | fungi: what about worker A? :) | 22:35 |
| *** radez is now known as radez_g0n3 | 22:36 | |
| fungi | mtreinish: my thoughts exactly | 22:36 |
| fungi | mtreinish: maybe it runs them in an arbitrary order | 22:36 |
| dims | clarkb, fungi: zuul jobs are definitely busted... any pointers? https://review.openstack.org/#/c/135434/ (i just changed a readme to illustrate) | 22:36 |
| *** shashankhegde has joined #openstack-infra | 22:37 | |
| clarkb | ianw: that was easy. approved | 22:37 |
| clarkb | dims: only tests.test_scheduler.TestScheduler.test_zuul_refs fails so that localizes the issue | 22:37 |
| *** baoli has quit IRC | 22:38 | |
| mtreinish | fungi: hmm, I just double checked it's complaining about logstash on a different thing than before | 22:38 |
| mtreinish | before it was: file { '/etc/logstash/subunit2sql.conf': | 22:38 |
| *** emagana has joined #openstack-infra | 22:38 | |
| *** baoli has joined #openstack-infra | 22:39 | |
| mtreinish | now it's: service { "jenkins-subunit-worker${suffix}": | 22:39 |
| clarkb | http://logs.openstack.org/34/135434/1/check/gate-zuul-python27/5d0897e/console.html#_2014-11-18_22_05_31_309 is where traceback stuff starts | 22:39 |
| *** mmaglana has joined #openstack-infra | 22:39 | |
| *** bhuvan has quit IRC | 22:39 | |
| clarkb | dims: I think its new version of GitPython | 22:41 |
| *** banix has quit IRC | 22:41 | |
| *** sarob has quit IRC | 22:41 | |
| clarkb | dims: want to pin to previous RC to see if that fixes? | 22:41 |
| dims | ah will try that clarkb | 22:41 |
| clarkb | looking at the traceback we expected a false value but got an exception instead soit looks like they changed an api somewhere | 22:41 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/zuul: EXPERIMENT: to see if jobs run properly https://review.openstack.org/135434 | 22:42 |
| mtreinish | fungi: do you think I should add an 'include logstash::indexer' for good measure? | 22:43 |
| dims | clarkb: if that's the problem, i'll adjust the test case and the usages? (or pin the version?) | 22:43 |
| fungi | mtreinish: so, weird, i edited the import line to import the required class rather than just the module's base class and the launch completed with no errors | 22:43 |
| fungi | mtreinish: but i still can't ssh into it | 22:43 |
| clarkb | dims: if thats the issue we should adjust the test case likely and pull up to require the latest version | 22:43 |
| clarkb | dims: however, we should probably test that well :) | 22:43 |
| fungi | mtreinish: i'm starting to wonder if we've done something bad[tm] to our puppet manifests | 22:43 |
| *** mmaglana has quit IRC | 22:44 | |
| *** yamamoto has joined #openstack-infra | 22:44 | |
| mtreinish | fungi: heh, do you think I'm breaking ssh? | 22:44 |
| mtreinish | fungi: yeah I'm wondering if I screwed up the policy around the ssh keys | 22:44 |
| *** jheroux has quit IRC | 22:44 | |
| dims | clarkb: ack. let me see how far i get | 22:44 |
| fungi | mtreinish: oh, where were you mangling that? | 22:44 |
| *** sarob has joined #openstack-infra | 22:44 | |
| clarkb | dims: actually we likely need to fix other code in zuul too | 22:44 |
| mtreinish | fungi: no | 22:45 |
| clarkb | dims: since we likely expect false and don't catch exceptions properly for new exceptions | 22:45 |
| mtreinish | fungi: or at least I don't think so: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/subunit_worker.pp | 22:45 |
| *** sputnik13 has quit IRC | 22:45 | |
| fungi | mtreinish: yeah, i didn't think you did. more than i want to try launching a plan generic instance and see if we've recently merged something which is breaking our account puppeting | 22:45 |
| *** packet has quit IRC | 22:45 | |
| *** vigneshvar_ has quit IRC | 22:46 | |
| *** kgiusti has left #openstack-infra | 22:46 | |
| *** sputnik13 has joined #openstack-infra | 22:47 | |
| *** sarob has quit IRC | 22:48 | |
| *** yamamoto has quit IRC | 22:49 | |
| *** emagana has quit IRC | 22:50 | |
| clarkb | fungi: mtreinish: anything I can help with? | 22:50 |
| *** baoli has quit IRC | 22:50 | |
| clarkb | remember puppet doesn't do things in order | 22:50 |
| *** emagana has joined #openstack-infra | 22:50 | |
| dims | clarkb: you were right on the money | 22:50 |
| clarkb | so if ou evaluate a require before the thing ou requie has fired well you lose | 22:50 |
| fungi | clarkb: not sure yet. so far consistently new nodes i launch prompt me for a password when trying to ssh. also prompts ssh as root from puppetmaster for a password | 22:50 |
| *** andreykurilin_ has quit IRC | 22:51 | |
| clarkb | wait what | 22:51 |
| *** krtaylor has quit IRC | 22:51 | |
| clarkb | have we touched anything ssh related in puppet? | 22:51 |
| * clarkb looks at logs | 22:51 | |
| *** baoli has joined #openstack-infra | 22:51 | |
| clarkb | fungi: oh unless, is puppet bailing out because it fails? | 22:51 |
| clarkb | and failing to apply ssh tings? | 22:51 |
| fungi | clarkb: maybe, though i didn't see any errors on the last run | 22:52 |
| fungi | clarkb: i'm trying with a randomly-named instance now, not mentioned in site.pp, just to see if it's any better | 22:53 |
| nibalizer | i would have blamed the forcing of uids... but if root is busted as well | 22:53 |
| clarkb | fungi: kk | 22:53 |
| *** eharney has quit IRC | 22:54 | |
| *** emagana has quit IRC | 22:55 | |
| *** bhuvan has joined #openstack-infra | 22:55 | |
| fungi | nope, randomly-named instance is working fine, so it's something about the specific manifests being applied to node /^subunit-worker\d+\.openstack\.org$/ | 22:56 |
| fungi | i'll see if i can narrow it down | 22:56 |
| mtreinish | fungi: heh, I knew it would come back to me :) | 22:57 |
| *** baoli has quit IRC | 22:57 | |
| fungi | all your fault mtreinish. i don't know how, just know that it is | 22:57 |
| fungi | _somehow_ | 22:57 |
| dims | lol :) | 22:57 |
| *** baoli has joined #openstack-infra | 22:57 | |
| mtreinish | fungi: that's how it always is :) | 22:58 |
| *** dkranz has quit IRC | 22:58 | |
| *** harlowja is now known as harlowja_away | 22:58 | |
| *** amitgandhinz has quit IRC | 22:58 | |
| *** weshay has quit IRC | 22:59 | |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Publish python-barbicanclient docs https://review.openstack.org/135449 | 22:59 |
| *** jistr|mtgs has quit IRC | 22:59 | |
| *** otter768 has joined #openstack-infra | 22:59 | |
| *** baoli has quit IRC | 23:00 | |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Publish python-barbicanclient docs https://review.openstack.org/135449 | 23:00 |
| *** baoli has joined #openstack-infra | 23:00 | |
| fungi | oh, you know what? it _could_ be something silently wrong with how my puppet dev env is set up | 23:00 |
| fungi | because when i tested the "include logstash" patch earlier, i did it with my dev env, and puppet didn't throw any errors | 23:01 |
| fungi | but now that the same patch is in master, testing with the production env throws the error about a missing dependency | 23:02 |
| clarkb | fungi: feel free to use mine | 23:02 |
| *** tsg has joined #openstack-infra | 23:02 | |
| *** emagana has joined #openstack-infra | 23:03 | |
| *** marun has quit IRC | 23:03 | |
| fungi | oh! | 23:03 |
| *** dprince has quit IRC | 23:03 | |
| fungi | manifest and modpath are wrong | 23:03 |
| fungi | clarkb: i think yours is broken too | 23:03 |
| clarkb | cool | 23:03 |
| *** krtaylor has joined #openstack-infra | 23:04 | |
| clarkb | I think I copied jeblairs :) | 23:04 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Adding trailing comma https://review.openstack.org/135450 | 23:04 |
| clarkb | so likely they are all wrong in that case | 23:04 |
| fungi | clarkb: yeah, your environment.conf refers to /opt/config/clarkb/... | 23:04 |
| nibalizer | fungi: nothing looks imediately wrong with the subunit node def or role class | 23:04 |
| fungi | nibalizer: yeah, now i'm suspecting a problem between my chair and the dev env i was using | 23:04 |
| *** jgrimm is now known as zz_jgrimm | 23:05 | |
| nibalizer | heh | 23:05 |
| clarkb | fungi: doesn't that exist? | 23:05 |
| * clarkb gets on puppetmaster | 23:05 | |
| fungi | clarkb: should be "system-config" now | 23:05 |
| *** baoli has quit IRC | 23:05 | |
| clarkb | oh we renamed it | 23:06 |
| clarkb | ya | 23:06 |
| clarkb | derp | 23:06 |
| *** baoli has joined #openstack-infra | 23:06 | |
| clarkb | and those files are owned by old clarkb | 23:06 |
| fungi | clarkb: the most marvellous thing about that is that puppet will happily apply nothing and not error out | 23:06 |
| clarkb | I will clean up my env | 23:06 |
| mtreinish | fungi: well nothing failed :) | 23:07 |
| fungi | mtreinish: indeed. nothing succeeded too | 23:07 |
| *** salv-orlando has quit IRC | 23:07 | |
| clarkb | ok I think my env is fixed assuming all I needed to do was edit that file, chown everything to my new uid in /etc/puppet/environments/clarkb and /opt/system-config/clarkb | 23:08 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Update module issues_url & include storyboard https://review.openstack.org/135452 | 23:09 |
| fungi | well, something's still wrong with my dev env even after fixing that | 23:11 |
| *** ChuckC has quit IRC | 23:11 | |
| *** ChanServ has quit IRC | 23:12 | |
| *** dangers is now known as dangers_away | 23:12 | |
| fungi | adjusted my environment.conf to look like the one for the production environment now... seeing if that's any better | 23:14 |
| *** baoli has quit IRC | 23:14 | |
| *** ChanServ has joined #openstack-infra | 23:15 | |
| *** sendak.freenode.net sets mode: +o ChanServ | 23:15 | |
| *** baoli has joined #openstack-infra | 23:15 | |
| *** banix has joined #openstack-infra | 23:16 | |
| *** salv-orlando has joined #openstack-infra | 23:16 | |
| fungi | okay, now it's actually applying the manifests i expected | 23:16 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/jenkins-job-builder: Make the recipients argument optional https://review.openstack.org/135455 | 23:16 |
| fungi | clarkb: you may want to adjust yours as well | 23:16 |
| clarkb | mriedem: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/elastic_recheck/manifests/init.pp#n26 | 23:17 |
| mtreinish | fungi: cool | 23:17 |
| clarkb | fungi: I thought I did | 23:17 |
| *** cnesa has quit IRC | 23:17 | |
| fungi | clarkb: i think it's the module path which was killing me | 23:17 |
| clarkb | mriedem: sorry I mixed up channels | 23:17 |
| mriedem | clarkb: that's fine, got the link, thanks | 23:17 |
| fungi | clarkb: just changing config to system-config wasn't sufficient. when i compared to the production env i saw the module path is listed in a different way now | 23:18 |
| clarkb | ah /me looks | 23:18 |
| clarkb | fungi: mine looks the same but I have flipped orders | 23:19 |
| clarkb | which is what I want so that my things apply before the "global" things | 23:19 |
| clarkb | fungi: am I missing something? | 23:19 |
| fungi | clarkb: dunno, but i was getting no modules applied at all except stdlib until i changed the order | 23:20 |
| fungi | and now it works | 23:20 |
| clarkb | huh | 23:20 |
| clarkb | ok I will change it | 23:20 |
| fungi | mtreinish: and it does appear you need to include the specific class, not just the module base class | 23:21 |
| fungi | mtreinish: with that adjustment, puppeting completes | 23:21 |
| *** liam__ has joined #openstack-infra | 23:21 | |
| *** baoli has quit IRC | 23:21 | |
| *** harlowja_away is now known as harlowja | 23:21 | |
| *** zul has quit IRC | 23:21 | |
| *** liam__ has quit IRC | 23:21 | |
| mtreinish | fungi: ok I'll push a patch out (unless you want the honors) | 23:21 |
| fungi | mtreinish: nah, have at it | 23:21 |
| *** baoli has joined #openstack-infra | 23:21 | |
| mtreinish | fungi: just s/logstash/logstash::indexer/ ? | 23:22 |
| fungi | i've put the worker in dns now too | 23:22 |
| fungi | mtreinish: yep | 23:22 |
| *** otherwiseguy has quit IRC | 23:22 | |
| *** salv-orlando has quit IRC | 23:22 | |
| *** ddieterly has quit IRC | 23:22 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Use correct include in worker.pp https://review.openstack.org/135456 | 23:23 |
| mtreinish | fungi: ^^^ | 23:23 |
| clarkb | fungi is it pulling things off the queue? | 23:23 |
| fungi | clarkb: not sure yet | 23:23 |
| clarkb | oh fyi I will be afk for much of tomorrow morning | 23:23 |
| fungi | k | 23:24 |
| *** salv-orlando has joined #openstack-infra | 23:25 | |
| mtreinish | fungi, clarkb: is there anything I can stare at to help check if things are working? | 23:27 |
| clarkb | mtreinish: not sure how much of it we have exposed. you don't have db access right? | 23:28 |
| fungi | mtreinish: i'm poking around in the instance right now to look for files i expect that class to install. some are there, some are not | 23:28 |
| fungi | mtreinish: for example, no initscripts | 23:28 |
| mtreinish | clarkb: nope, it's also trove so it's on the private network... | 23:28 |
| clarkb | mtreinish: "private" network | 23:28 |
| fungi | mtreinish: subunit2sql.conf is in /etc/logstash/ but i see no worker yaml files in that directory | 23:29 |
| *** unicell has quit IRC | 23:30 | |
| fungi | subunit2sql is installed according to pip list, but /usr/local/bin/subunit-gearman-worker.py is not present | 23:30 |
| mtreinish | fungi: hmm, that's definitely in the worker.pp | 23:30 |
| fungi | yet both of those are in the base class | 23:30 |
| *** dims_ has joined #openstack-infra | 23:31 | |
| fungi | i think maybe not all of the manifest for this was applied. i'm going to try updating puppet on it in just a moment and see if it leaps into action | 23:31 |
| *** oda-g_ has quit IRC | 23:31 | |
| mtreinish | oh, those are things that get pulled in from puppet:// could it be something else with your dev env? | 23:31 |
| fungi | that's possible | 23:32 |
| *** banix has quit IRC | 23:32 | |
| fungi | though if clarkb is cool with approving 135456 i can just try updating from the production env now | 23:33 |
| * clarkb looks | 23:34 | |
| *** dims has quit IRC | 23:34 | |
| clarkb | well wait | 23:34 |
| clarkb | do ou need both of those things to be included? | 23:34 |
| clarkb | puppet does not give you logstash if ou include logstash::indexer (it isn't pthon) | 23:34 |
| clarkb | do the required files from from logstash? | 23:35 |
| fungi | the only logstash resource i see there is logstash::indexer | 23:35 |
| mtreinish | clarkb: IIRC all that is really needed is that the /etc/logstash dir is created | 23:35 |
| mtreinish | because that's aparrently where I decided to dump everything... | 23:36 |
| *** dimtruck is now known as zz_dimtruck | 23:36 | |
| clarkb | oh ok logstash::indexer includes logstash | 23:36 |
| clarkb | so in this case it will do the right thing | 23:36 |
| clarkb | except | 23:36 |
| clarkb | that will run a logstash indexer | 23:36 |
| clarkb | which we don't want | 23:36 |
| clarkb | meh firewalls should prevent that from screwing anything up | 23:37 |
| clarkb | we can clean it up later | 23:37 |
| fungi | any way to just depend on (and if necessary) provide /etc/logstash in the worker.pp? | 23:37 |
| clarkb | I approved the change | 23:37 |
| *** ChuckC has joined #openstack-infra | 23:37 | |
| *** sarob has joined #openstack-infra | 23:37 | |
| clarkb | feel free to -2 if you don't like what that may do | 23:37 |
| fungi | nah, it's likely safe for now | 23:37 |
| fungi | was asking more as a cleanup to do | 23:38 |
| clarkb | ya I think you can just have a file resource in there | 23:38 |
| clarkb | that makes /etc/logstash a dir | 23:38 |
| fungi | depending on the entire class to just get a directory seems heavy | 23:38 |
| clarkb | especially when that brings in a service | 23:39 |
| mtreinish | yeah, sloppy, only partially functional, policy is what happens when I write it... :) | 23:39 |
| mtreinish | this is the first time I've touched puppet in like a year... | 23:39 |
| *** thedodd has quit IRC | 23:40 | |
| mtreinish | fungi, clarkb: I can push a patch to clean it up, one sec | 23:40 |
| *** mmaglana has joined #openstack-infra | 23:40 | |
| fungi | the main thing is that once that patch merges, we should be able to get puppet applying from the puppetmaster and be able to see the effects of any subsequent changes | 23:41 |
| clarkb | ya | 23:41 |
| *** sarob has quit IRC | 23:42 | |
| fungi | this is just at that ugly bootstrapping phase where we have to get it into good enough shape where it requires less involvement from us to continue improving it | 23:42 |
| *** sarob has joined #openstack-infra | 23:42 | |
| JayF | Thanks everyone for the help; we just got all the IPA CI (including tempest when building a new agent version) passing | 23:44 |
| openstackgerrit | Merged openstack-infra/system-config: Use correct include in worker.pp https://review.openstack.org/135456 | 23:45 |
| *** mmaglana has quit IRC | 23:45 | |
| fungi | JayF: congrats--that's exciting! | 23:45 |
| *** unicell has joined #openstack-infra | 23:46 | |
| JayF | it really is, | 23:46 |
| JayF | I'm curious what all it should vote on | 23:46 |
| JayF | IPA/Ironic is clear, and we discussed that in project | 23:46 |
| *** alexandra_ has joined #openstack-infra | 23:46 | |
| JayF | but what about Devstack? Tempest? the other sets of things that can break things? | 23:46 |
| *** xyang0 has quit IRC | 23:46 | |
| *** gokrokve has joined #openstack-infra | 23:47 | |
| fungi | JayF: i guess the question is whether those are things where a mistake is likely to break only ipa, or everything | 23:47 |
| mtreinish | JayF: is it a 3rd party ci? (sorry I haven't been following what you're doing) | 23:47 |
| mtreinish | JayF: but, yeah what fungi said | 23:47 |
| clarkb | no they wanted third party ci and I yelled at them >_> | 23:47 |
| JayF | and we did it the right way instead | 23:47 |
| fungi | mtreinish: the ironic python agent tests | 23:47 |
| JayF | yay us and clarkb | 23:47 |
| JayF | lol | 23:47 |
| JayF | fungi: I'd suspect it's *very easy* for a devstack change to break building the IPA image | 23:48 |
| mtreinish | fungi: ah, ok | 23:48 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/zuul: EXPERIMENT: to see if jobs run properly https://review.openstack.org/135434 | 23:48 |
| JayF | fungi: for tempest; I don't think it matters post-testing split (i.e. when ironic jobs are in ironic), but I'd be concerned someone would make a change that passes pxe_ssh job but not agent_ssh job | 23:48 |
| JayF | especially since we're configured differently until agent job can be run in parallel (Which basically boils down to needing a lower ram footprint) | 23:48 |
| *** asettle has quit IRC | 23:49 | |
| *** alexandra_ is now known as asettle | 23:49 | |
| fungi | JayF: so if that's the case for devstack, you probably want to appeal to the devstack developers for ways to mitigate that (whether it's co-gating on that job or some other lighter-weight confirmation on the behaviors which are important to ipa) | 23:49 |
| *** markmcclain has quit IRC | 23:49 | |
| JayF | fungi: like a lighter job that would just make sure it built something that looks like an ipa ramdisk/kernel without running the whole tempest setup? | 23:50 |
| fungi | JayF: maybe | 23:50 |
| fungi | JayF: not sure what it would look like, hence the suggestion for brainstorming with them | 23:50 |
| JayF | OK, but for starters, Ironic+IPA and nothing else, then maybe ask this question on-list? | 23:50 |
| fungi | JayF: yeah (assuming devananda is okay with gating ironic on that) | 23:51 |
| fungi | presumably he'll be in favor | 23:51 |
| devananda | JayF: one thing to keep in mind is transitive dependencies ... A+B, B+C => A+C | 23:51 |
| JayF | devananda: wdym? | 23:52 |
| devananda | fungi: I don't recall - is there a way to avoid that today? | 23:52 |
| devananda | IPA+Ironic, Ironic+Nova => IPA + Nova ... | 23:52 |
| JayF | fungi: devananda: we talked a lot about testing matrices for IPA at the mid-cycle last time. Idea was basically IPA would check against all possible images (DIB/CoreOS), Ironic would check IPA with one image (probably DIB), and then we rely on Ironic's driver model to "hide" agent differences from us, so other things (like nova) only check against default driver (today: pxe_ssh) | 23:52 |
| fungi | devananda: transitive relationships between projects sharing jobs are mostly only relevant to whether they appear in the same dependent change queue | 23:53 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Remove unnecessary use of logstash::indexer from subunit_worker https://review.openstack.org/135463 | 23:53 |
| *** doug-fish has joined #openstack-infra | 23:53 | |
| JayF | devananda: ^ that's a reasonable approximation of what we talked about in Oregon, right? | 23:53 |
| devananda | fungi: "dependent change queue" does not mean anything to me at the moment | 23:53 |
| mtreinish | fungi, clarkb: ^^^ I wonder if I should change it to /etc/subunit or something? | 23:53 |
| devananda | probably because I just got off the phone | 23:53 |
| fungi | devananda: whether the changes get sequenced together in the gate or get to have their own separate line to gating | 23:53 |
| mtreinish | I'm not exactly sure what the spam in openstack-meeting is supposed to be | 23:54 |
| fungi | ooh, spam! | 23:54 |
| *** emagana has quit IRC | 23:54 | |
| * JayF has a primary goal of making sure IPA's jobs don't break after everyone ( jroll and many others) worked so hard to get them passing | 23:54 | |
| fungi | clarkb's on the ball | 23:55 |
| *** emagana has joined #openstack-infra | 23:55 | |
| clarkb | well we'll see ho w effective those kickbans are | 23:55 |
| mtreinish | oh, I got it now, I had to resize my terminal... (it's about what I expected) | 23:55 |
| clarkb | users joined with webclients | 23:55 |
| clarkb | and may just rejoin and I will need a more fanciful mask | 23:55 |
| devananda | JayF: that is an accurate approximation, yes. and I still agree with it :) | 23:55 |
| *** amotoki_ has joined #openstack-infra | 23:56 | |
| *** doug-fish1 has quit IRC | 23:56 | |
| * JayF amends his PRs to make Ironic and IPA vote on that job | 23:56 | |
| JayF | we can merge them whenever we feel the job is stable enough | 23:56 |
| devananda | fungi: if the changes JayF are proposing will "have their own separate line to gating" then yes | 23:56 |
| clarkb | I may have banned anyone using the webclient <_< | 23:56 |
| clarkb | but whatever | 23:56 |
| JayF | devananda: i have no idea what that means, lol | 23:56 |
| clarkb | fungi: ^ is that something I can make less terrible? | 23:56 |
| *** mriedem has quit IRC | 23:56 | |
| mtreinish | clarkb: heh, I think that's fine :) | 23:57 |
| devananda | JayF: i'm still catching up on things -- is there a graph of the stability of the IPA jobs up already? | 23:57 |
| *** amotoki has quit IRC | 23:57 | |
| *** banix has joined #openstack-infra | 23:57 | |
| *** boris-42 has quit IRC | 23:57 | |
| *** alexandra_ has joined #openstack-infra | 23:57 | |
| JayF | devananda: the agent_ssh job has been passing for ~2 weeks now; the agent_ssh-src (where the agent is built on the fly to test IPA changes) job just started passing about 30 minutes ago, lol | 23:57 |
| *** alexandra_ has quit IRC | 23:57 | |
| devananda | \o/ | 23:58 |
| *** alex7376 has joined #openstack-infra | 23:58 | |
| JayF | devananda: agent_ssh IMO already was stable enough to vote; but I didn't want anything to vote until IPA could have that job voting too | 23:58 |
| fungi | clarkb: i'm not too concerned about the kiwiirc.com web client failing to work for openstack meetings | 23:58 |
| JayF | for obvious reasons | 23:58 |
| jroll | devananda: adam_g has the agent_ssh check for ironic in his thing | 23:58 |
| clarkb | fungi: scrollup I got freenode web client with a larger glob | 23:58 |
| clarkb | I guess I can ban without that extra glob then remove glob ban /me does that | 23:58 |
| devananda | so we want to test, essentially, on any change in Ironic: stable-IPA + tip-Ironic; and on any change in IPA: tip-IPA + stable-Ironic. | 23:58 |
| devananda | for some nebulous definition of "stable" | 23:58 |
| clarkb | on the second one I realized that I may have derped all webclients from freenode :) | 23:58 |
| *** lttrl has quit IRC | 23:59 | |
| devananda | JayF: that sound right? | 23:59 |
| fungi | clarkb: meh, okay mostly the same opinion though. if i had to find new things to care about, web irc client access to openstack meetings is pretty far down the candidate list | 23:59 |
| *** emagana has quit IRC | 23:59 | |
| devananda | JayF: or am I derailing | 23:59 |
| clarkb | I will clean it up a bit because I can do this better :) | 23:59 |
| *** gokrokve has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!