Saturday, 2016-03-05

mordrednibalizer: which is that ansible users who happen to upgrade python-neutronclient are completely broken right now00:00
mordrednibalizer: maybe I should have led with that00:00
*** rbrndt has quit IRC00:00
clarkbError: Could not stat device /dev/vdc - No such file or directory. the mystery deepens00:00
mordrednibalizer: we had a bug in deserialization that got tickled by them changing something slightly in their output format00:00
nibalizerbecause of that 'to-munch-or-not-to-munch patch?'00:00
mordrednibalizer: not really- although it is munch related00:01
*** kzaitsev_mb has joined #openstack-infra00:01
mordrednibalizer: it's that we had things doing if type(obj) == dict ... rather than isinstance(obj, dict)00:01
mordrednibalizer: which means when they started returning a dict subclass instead of a dict00:01
nibalizeryea thats what i was talking about00:01
mordredwe went boom00:01
nibalizerperhaps to obiqueley00:01
mordredyeah. then YES00:01
mordredthere is another patch that discusses whether or not we shoudl really be munching00:01
jeblairi like how you are almost communicating with each other :)00:01
*** dbrycen has quit IRC00:02
jeblairit's friday beer time clearly00:02
*** ddieterly has quit IRC00:02
bkeroBy beer you mean deploy to production00:02
clarkbhrm --keep isn't useful because I don't have an ssh key to log in with yet00:03
clarkbsilly clouds00:03
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add zlib1g-dev to bindep-fallback.txt
* clarkb finds the private key00:03
nibalizermordred: when did they python-neutronclinet release happen?00:03
fungiclarkb: pabelanger: ^ so far that's the only one i find missing going back through modules/openstack_project/manifests/jenkins_params.pp additions as far as 1.5 years (just to be safe)00:03
fungiso it was really just the one change which slipped through adding libjpeg-dev and zlib1g-dev00:04
pabelangerfungi: WFM00:04
pabelangerfungi: ya, that is pretty good00:04
mordrednibalizer: this week00:05
mordrednibalizer: like 2 days ago?00:05
fungialso the state of the bindep fallback list has gotten pretty terrible. i'm going to do a cleanup pass on it (references to platforms we don't run or being unnecessarily specific on platform matches, or making redundant entries... it's well longer than it needs to be)00:05
*** e0ne has quit IRC00:05
nibalizermordred: how long has it been since a shade release?00:05
mordrednibalizer: that long00:06
*** ashleighfarnham has quit IRC00:06
mordredShrews: we should maybe release more frequently00:06
nibalizermordred: yea that does not look like a friday release00:07
*** rguillebert has quit IRC00:07
*** annegentle has quit IRC00:07
*** ashleighfarnham has joined #openstack-infra00:07
nibalizerif you want to make a branch and cherry pick the bare minimum and cut a 1.4.1 im cool with it though00:07
clarkbactually looks like maybe my lvm2 intssall failed I think their imgae must be specialer00:08
* clarkb uploads an iage00:08
nibalizerlooking at the diff | wc -l is 4k lines00:08
jeblairclarkb: are we also waiting on the swift changes pending a shade release?00:09
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Sort bindep-fallback.txt
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Clean up bindep-fallback.txt
fungipabelanger: ^00:09
mordredjeblair: ya. the swift changes are also pending a shade release00:10
*** Jeffrey4l has quit IRC00:10
fungiso not as terrible as i thought00:10
Shrewsmordred: i've been nagging about releasing. perhaps you should teach me release stuff at some point?00:11
nibalizerShrews: its git tag 'tag'; git push --tags00:12
*** thorst has joined #openstack-infra00:12
pabelangerfungi: I am not sure liberasurecode-devel is a thing on centos00:12
nibalizerthere is video of mordred doing it at his real slim shade talk in tokyo00:12
pabelangerfungi: but gate will tell us00:12
jeblairnibalizer, Shrews: "git tag -s <tag>"00:12
fungipabelanger: well, it was being applied to centos-7 previously00:12
nibalizerah yes, that00:12
clarkbI am uploading the same image that I used in vexxhost to osic00:12
jeblairdon't forget the -s00:12
nibalizerya it needs the s00:12
clarkbto avoid debconf being unhappy when trying to apt-get install -y00:12
nibalizernot your -s, but just any old -s00:12
openstackgerritMerged openstack-infra/system-config: Use osic-cloud1
fungipabelanger: that change just condenses fedora22+fedora23+centos7 to rpm00:13
Shrewsnibalizer: jeblair: by 'learn' i mean, give me permission and let me know if there are release rules00:13
jeblairnibalizer: though i will lend mine if needed00:13
pabelangerfungi: Oh, ya.00:13
nibalizermtreinish: did we get the subunit worker thing?00:13
pabelangerfungi: guess it is time to call it a day00:13
* nibalizer goes to look00:13
Shrewsi know tagging does the magic00:13
funginibalizer: also git push --tags is dangerous00:14
funginibalizer: push a specific tag by name00:14
funginibalizer: we have the workflow documented!
nibalizeris that with git push <tagname> ?00:14
pabelangerhave a good weekend all00:14
fungigood weekend pabelanger!00:15
jeblairShrews, mordred: ah that's fungi's department00:15
jeblairpabelanger: o/00:15
clarkbif I get this mirror built soonish are we going to make nodepool go? we totally should :)00:15
jeblairclarkb: ++00:15
jeblaircody-somerville: i'm trying out your change :).  apparently the font i'm using does not have the unicode clock, though it does have the triangle.00:16
cody-somervillejeblair: doh. What font are you using?00:17
fungiShrews: i'm not opposed to you having shade-release membership if you've been wanting to release it and getting up with someone to do it has been getting in the way of progress. it's mostly that we keep the -release groups constrained because if you don't exercise the skill often, you have a higher chance of creating a broken release (which is sometimes worse than not releasing often)00:17
clarkbapparently when we fail to launch node the cleanup doesn't cleanup floating ips00:17
* clarkb manually cleans those out00:17
mordredclarkb: oh nice00:18
mordredclarkb: we have a flag for that in shade00:18
clarkbya  Ithink it just needs updating00:18
clarkbsince it is the shade launch node00:18
mordredclarkb: you can just say "cloud.delete_server(server_name, delete_ips=True)00:18
fungijeblair: most terminals will composit glyphs for missing codepoints from other fontsets you have installed00:18
mordredclarkb: the script says delete_ips=True00:19
mordredthis does not happy me00:19
fungijeblair: so your preferred font doesn't need to have it as long as you have a font which has it (though that behavior may depend on your font libraries and config)00:19
mordredShrews: not sure if you know about launch/ in system-config00:19
clarkbubuntu provided image totally works so the one provided by osic has a different debconf for some reason00:19
mordredShrews: but clark seems to have found a place where it's not doing what we expect00:20
Shrewsfungi: i only offer to take load off of mordred. i don't care either way00:20
clarkboh I take that back what is going on00:20
Shrewsmordred: sad making00:20
Shrewsshould i pour another scotch before here said sad causing thing?00:20
mordredShrews: you should always pour scotch00:21
mordredShrews: or japanese whiskey00:21
Shrewsmmm, nikka00:21
mordredheck yes00:21
Clintcorrect japanese whiskey00:21
jeblaircody-somerville, fungi: i'm using 6x13.00:21
fungiShrews: mordred: i've personally avoided trying to release shade myself because i've not had time to figure out if i'm doing the things reno wants of me, or simply preparing to anger it00:21
clarkbmordred: its probably because the create works but then later we explode and don't roll back00:21
clarkbmordred: because we do things like attempt to mount the cinder volume and fail00:22
clarkbbceause there is no /dev/vdc despite asking nova/cinder to attach at that device00:22
jeblair"urxvt -fn 6x13"00:22
mordredfungi: we do not use reno in shade the way other things do00:22
mordredfungi: so you're safe00:22
fungimordred: er, even more daunting?00:22
mordredclarkb: but - the except does a delete_server00:22
fungihow do you use reno then?00:23
mordredfungi: we put it in the normal sphinx docs00:23
mordredfungi: becuase we do not publish to releasenotes.o.o00:23
mordredsince that would be weird00:23
clarkbmordred: oh hrm I dunno then00:23
jeblairmordred: what's ?00:23
jeblairi'm having an argument with dns about its existence00:24
mordredor, whereever they go00:24
fungioh. is this one of the libraries that's annoying package maintainers because of reno scraping git history when they try to generate project documentation?00:24
jeblairi'm losing00:24
fungijeblair: might be releases.openstack.org00:24
Shrewsjeblair: try saying bad things about dns's hands00:24
clarkbI am trying again without specifying the device00:24
Shrewsjeblair: b/c that seems to work in GOP debates00:24
jeblairShrews: i think i will lose harder00:25
fungijeblair: actually i don't know if bitmap fonts will have a similar behavior. i'm using truetype rendering since i wanted fonts with good east asian codepoint coverage00:25
clarkbnope config drive really does appear to be at vdb00:25
jeblairmordred, fungi: it looks like releases.o.o is what mordred meant, but the actual release notes end up at eg:
clarkbI wonder if cinder isn't working here00:25
clarkbor cinder + nova00:25
clarkbI am going to try manually00:26
jeblairfungi: yeah, i use bitmap fonts in terminals; i have a 600dpi threshold for tolerance of antialiasing (which is, incidentally, and totally coincidentally, pretty close to when you don't need it anymore)00:27
Shrewsclarkb: mordred: i noticed the recent release of cinder is more strict when deleting things. i.e., we first have to delete snapshots before deleting the volume (unless using a new --cascade option). wondering if the ip thing is similar00:27
Shrews(if i grok the scrollback correctly)00:28
mtreinishnibalizer: I don't think so, the puppet patch still needs a +A00:28
mtreinishafter that I think it should be good to go00:28
mordredShrews: ah. so we might be failing in the delete00:29
cody-somervillejeblair: Hmm... you might be able to upgrade your font. It looks like 6x13 (misc-fixed) supports CEN MES-3A European Unicode 3.2 Subset which should have it.00:29
nibalizerany other cores want to look at
fungiat some point in the past i discovered that certain sans-serif truetype fonts are easier on my dyslexia and actually work well monospaced, or else i too would probably still be using traditional x11 bitmap fonts00:30
clarkbmordred: Shrews fwiw the last couple of deletes seemed to work so maybe I was derping00:31
cody-somervillejeblair: The symbol I used is from Unicode
clarkbI only kept one server but had 3 leaked fips so don't know how that happened00:31
*** akshai has joined #openstack-infra00:31
mordredclarkb, Shrews: so - honestly, the next amount of work I want to do there is to get us from shade-launch-node to ansible-launch-node00:31
*** kingia has joined #openstack-infra00:32
fungiclarkb: are you sure those "leaked: fips weren't already in there from something else?00:32
clarkbfungi: pretty sure, I hadn't used this account (had used jenkins instead of ci)00:33
*** akshai has quit IRC00:33
clarkbok osic puts config drive on /dev/sr000:33
clarkbso not the vd* conflict00:33
fungii mean, it's possible osic launched something to test the account, then deleted but forgot to clean up the fip00:33
clarkbsomething else funny is happening00:33
clarkbfungi: oh maybe00:33
*** kingia_ has quit IRC00:35
*** dimsum_ has quit IRC00:35
clarkbwriting this here so I don't forget, it would be nice if clouds set their default compute az to the same location as their default else az so that I don't have to constantly specify these things after the fact and rearrange00:37
clarkbok debugging has happened00:41
clarkb/dev/vdb works except we make_swap on that :/00:41
clarkbso I am just going to force attach to /dev/vdc to bypass that >_>00:41
mordredclarkb: ++00:42
mordred(to the az thing)00:42
cody-somervillejeblair: Just downloaded the latest from and our symbol doesn't show up in the missing characters file for 6x13.00:42
*** dingyichen has quit IRC00:42
clarkband that still doesn't work00:43
*** sigmavirus24 is now known as sigmavirus24_awa00:43
*** Swami has quit IRC00:43
clarkbits almost like specifying the device path isn't a thing00:43
clarkbI am going to ninja around this by just not make swapping00:43
*** Swami_ has quit IRC00:43
mordredcody-somerville: there was a symbol?00:43
mordredclarkb: I'm about to EOD - however, the thing you're doing right now is a thing I would like for shade/ansible to be able to help with00:44
mordredclarkb: so when you're done, would you send me a short whatever about what went on?00:44
clarkbyup, though relaly there are a couple cloud side issues here00:44
clarkb1) config drive and cinder volumes can fight over a device and 2) specifying device doesn't work?00:45
mordredI just want to make sure that shade fixes as manyof them as it can00:45
mordred1) WHAT?00:45
mordred1) WHAT?00:45
*** markvoelker has quit IRC00:45
clarkbmordred: yes basically config drive can go on /dev/vdb and cinder/nova think the volume goes there too00:45
clarkbmordred: the best part is when you reboot the config driv emoves to /dev/vdc and volume goes on /dev/vdb00:46
mordredclarkb: oh. SO...00:46
mordredclarkb: there is a thing that might be useful00:46
mordredclarkb: which is that everywhere except xen (and I think BobBall_AWOL fixed xen)00:46
mordredclarkb: the name you give a cinder volume will show up in the fancy device name volume stuff00:47
clarkbok got past that particular issue, we might have a mirror in a few minutes00:47
clarkbmordred: under /dev/disk/by-lable?00:47
cody-somervillemordred: Used in my patch to Gertty. I added navigation breadcrumb, see and for example see
mordredclarkb: yes00:47
mordredclarkb: so if you do "cinder create foo"00:47
mordredclarkb: you sohuld e able to get /dev/disk/by-label/foo00:47
clarkbmordred: that could be helpful except that a cloud shouldn't do the thing it is doing :/00:47
clarkbwhich is get confused00:47
mordredclarkb: dude. if a cloud ever did something right ...00:48
clarkbok this seems to have worked by not running make swap00:48
clarkband letting cinder/nova assignt the device00:48
clarkbso I have that bit of feedback for osic "cannot specify dvice with cinder attach"00:48
clarkbbtw with all these flavors that don't have an ephemeral disk /dev/vdb is actually something we probably don't want swap on00:50
*** Daisy has joined #openstack-infra00:50
clarkbwe might fix that by specifying an explicit swap device?00:50
clarkbI am adding dns now00:51
mordredclarkb: woot!00:51
clarkbactually let me check disk setup first00:51
clarkblooks good adding dns now00:52
*** ccrouch1 has quit IRC00:52
clarkb should be working shortly00:53
clarkbonce it does any opposition to me approving the thing to add to nodepool?00:53
*** ccrouch has joined #openstack-infra00:53
mordredclarkb: nope00:53
clarkb8.8.8.8 resolves it, I am going to approve nodepool add now00:54
clarkbwow jenkins02 is still going00:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
*** roxanaghe has quit IRC00:55
rcarrillocruzShrews: ^00:56
Shrewsrcarrillocruz: may i offer an alternative?00:58
*** Daisy_ has joined #openstack-infra00:58
mordredheh. I just accidentally approved a patch in gertty because I thought I was switching to IRC window #3 (A-3) but instead I was in gerrty, so (A-2) approve00:59
*** Daisy_ has quit IRC00:59
mordredluckily, it was a patch that I'd already reviewed and jeblair had already approved :)00:59
*** Daisy_ has joined #openstack-infra00:59
Shrewsrcarrillocruz: instead of a new param, why not just always do a get_domain() with whatever domain_id is. then document that domain_id can be either name or id00:59
bswartzhey guys, I have a question about he "scp" publisher in jenkins -- it seems that it directly overwrites existing files, and for the time during the upload, anyone who downloads the file gets something corrupted -- would it be possible to upload to a temp file and then atomically rename the temp file to the permanent name? does jenkins allow that?01:00
bkeromordred: Should invent a way to sign your approvals with your gpg key, that way you need to type your passphrase to do it01:00
bkerono accidental approvals :)01:00
jeblairwell, also, you have like half an hour to change your mind and revoke the approval :)01:00
clarkbbswartz: not with the scp plugin01:00
clarkbbswartz: there are alternatives like using rsync or uploading to swift01:00
rcarrillocruzShrews: i think that'd be more confusing to shade users. Having domain_id a meaning of *domain_id* in all places but on these operations name or id is confusing01:01
jeblairbkero: then i could have gertty cache the passphrase for convenience!01:01
*** Daisy has quit IRC01:01
bkerojeblair: that sounds really convenient. I'll draw up a blueprint asap01:02
rcarrillocruzin other news spacex failed again01:02
Shrewsrcarrillocruz: i can see that. but it would also be more consistent with the other apis that accept name or id as the 1st arg01:02
Shrewsrcarrillocruz: just tossing out an idea for consideration01:03
fungibswartz: this is also a known issue with, e.g., branch tarball publication01:03
bswartzclarkb: can I use scp to upload the file and then something else afterwards to do the rename? "ssh mv <temp> <perm>" would work if jenkins can do that01:03
fungithere are times when if you try to retrieve it while it's in the process of being replaced, you get a truncated copy01:03
clarkbbswartz: I think if you go that route just bypass jenkins scp plugin entirely01:04
clarkband use eg rsync01:04
jeblairrcarrillocruz: the barge is called "Of Course I Still Love You"?  they are iain m banks fans!01:04
openstackgerritMerged openstack-infra/project-config: Use OSIC with nodepool
fungibswartz: the scp publisher plug-in in jenkins doesn't really do anything beyond invoke scp01:04
rcarrillocruzShrews: a 3rd one would be 'create_domain_now_with_name_or_id_yay(self, name_or_id, blah)01:04
rcarrillocruzjust kidding :D01:04
bswartzokay well the window of danger is fairly narrow01:04
Shrewsrcarrillocruz: YES! so do that01:04
* Shrews reaches internet tolerance level. logs off. night all01:05
fungiclarkb: bswartz: part of the challenge here is that the publishers run in the context of the jenkins master, which is why they're generally feature-poor. they don't run on the worker with access to the original workspace01:05
bswartzwe've observed uploads taking from 40-60 seconds, and if we do that on average once per week, the chance of a job getting bitten by that is vanishingly small01:05
*** gyee has quit IRC01:05
NobodyCamfungi: just checking ... good for that recheck?01:05
* rcarrillocruz goes to bed with the sweet flavor of green tests on his change, night all01:05
bswartzalso if it gets bad enough we can move our gate tests to rely on tagged qcow files01:06
*** sdake has joined #openstack-infra01:06
bswartzI just wondered if there was an easy way to fix this01:06
fungiNobodyCam: i have no idea what the status of the images is. clarkb: were you the one who kicked off the rebuilds?01:06
clarkbfungi: I did, everywhere but ovh-bhs one should be good01:06
clarkbexcept still no zlib01:07
*** Daisy_ has quit IRC01:07
*** Daisy has joined #openstack-infra01:08
fungibswartz: we've got a couple of ideas for improved publication mechanisms. our swift publisher can upload directly from the worker into a permission-constrained location in a swift container, but that needs special frontend care to index it. we also have jobs running on privileged workers that can stick files into afs and serve them from our mirror network, and a design not yet implemented for an01:08
fungiatomic publication pipeline for documentation relying on an afs volume01:08
*** mtanino has quit IRC01:09
* mordred points to the epic win of spinning up new mirror hosts we're experiencing01:09
bswartzfungi: as long as the image is ultimately downloadable with wget I don't think we care what goes on behind the scenes01:09
fungiclarkb: NobodyCam: 288805 adds zlib1g-dev and its rpm equivalent01:09
bswartzwhat you describe sounds like overkill for our simple use case though01:10
*** Qiming has joined #openstack-infra01:10
*** ashleighfarnham has quit IRC01:11
*** ashleighfarnham has joined #openstack-infra01:11
mordredyah. we're more working on the general problem of "how do jobs publish things"01:11
mordredhoping to solve it in such as way as that you can easily not care how it works ever01:12
jeblaircody-somerville: many of the breadcrumbs seem too verbose to be useful... i wonder if screens should have a 'short title'...?01:13
jeblaircody-somerville: so rather than "All changes in My ..." we would have "My changes".01:14
mordredjeblair: is this a gertty change?01:14
*** Daisy has quit IRC01:14
jeblairmordred: yeah, the one cody-somerville linked to earlier:
*** Daisy has joined #openstack-infra01:15
cody-somervillejeblair: The same idea occurred to me as well.01:15
mordredBlack right pointing small triangle01:15
*** ashleighfarnham has quit IRC01:15
mordredthat's such a better letter name than "epsilon"01:15
jeblairmordred: ya, i didn't even know you could do that.... << cody-somerville  :)01:15
*** bpokorny_ has joined #openstack-infra01:15
*** Daisy has quit IRC01:16
*** Daisy has joined #openstack-infra01:16
jeblair(might be a good followup change to experiment with that)01:17
*** bpokorny has quit IRC01:18
*** bpokorny_ has quit IRC01:20
*** Sukhdev has joined #openstack-infra01:21
*** pvaneck has quit IRC01:22
cody-somervillejeblair: Yup. Was also thinking about making the breadcrumb interactive in subsequent change - e.g. click breadcrumb to switch to that view01:24
cody-somervillejeblair: or be able to pan when there is overflow. might be able to do something like show more of the title if selected or something.01:25
*** ddieterly has joined #openstack-infra01:26
openstackgerritMerged openstack-infra/project-config: Add zlib1g-dev to bindep-fallback.txt
jeblaircody-somerville: yeah, clicking seems like a good next step01:28
*** davide85_ has quit IRC01:29
*** Daisy has quit IRC01:30
*** mtanino has joined #openstack-infra01:30
*** dims has joined #openstack-infra01:30
*** Daisy has joined #openstack-infra01:30
cody-somervillejeblair: fyi, in my testing I noticed that I could not get the TOP_SCREEN command to trigger. If I'm reading right, it should be alt+home key. I think it might be related to Unity binding the meta key. I did change the binding locally to test but figured I'd point it out since Ubuntu is popular.01:30
jeblaircody-somerville: ah, thanks.  indeed that's what it's supposed to be (wfm under xfce), and i can understand not having that would make navigation slightly cumbersome... i'm open to alternate suggestions.01:32
jeblaircody-somerville: er, wait, i think it's ctrl-home01:32
jeblaircody-somerville: oh weird01:33
fungii find it interesting that just home and end don't do top/bottom of screen01:33
cody-somervillejeblair: Says meta home. In any case, I tried all three keys down there with home and it didn't work. But changing it to say ctrl + p did.01:33
jeblaircody-somerville: it does say 'meta home' and when i hit 'ctrl home' it triggers it.01:33
jeblairi can not immediately explain that.01:34
jeblairfungi: yeah, they should, which is why i didn't bind top_screen to home...  this is probably fixable.01:34
cody-somervillejeblair: so yea, a second default keybinding might not be a bad idea.01:34
*** Daisy has quit IRC01:34
cody-somervillehome seems to be bound to CURSOR_MAX_LEFT01:35
cody-somervillewhich does seem to work as expected when you're entering text.01:35
cody-somervilleWe could definitely have it do top/bottom of screen when not entering text.01:36
fungithat would make sense to my finger memory01:37
cody-somerville(and we should find more excuses to add more unicode symbols, they're fun! lol)01:37
clarkbreally need to make nodepool builders reload their config01:39
*** yamahata has quit IRC01:39
*** mtanino has quit IRC01:40
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Only create 1024MB swap in tripleo-ci
*** sdake has quit IRC01:43
clarkbwaiting on the ovh bhs image upload to complete so I can restart the builder service01:43
*** rfolco has joined #openstack-infra01:44
*** Sukhdev has quit IRC01:45
*** markvoelker has joined #openstack-infra01:46
*** alivigni76 has joined #openstack-infra01:48
fungioh, also if mordred's nodepool debug logging patches landed, might also be a good time for a nodepoold restart01:49
*** alivigni76 is now known as _ari_01:49
fungispeaking of ovh01:49
*** _ari_ has quit IRC01:49
*** rfolco has quit IRC01:52
*** keedya has quit IRC01:54
clarkbI am tempted to stop the builder anyways01:55
clarkbsince we need a new image in bhs for zlib01:55
*** dims has quit IRC01:55
NobodyCamyep error has changed to : ValueError: jpeg is required unless explicitly disabled using --disable-jpeg, aborting01:55
*** keedya has joined #openstack-infra01:56
clarkbNobodyCam: jpg is present on the image snow except for in ovh-bhs101:56
fungiclarkb: i suppose if ovh is taking too long to update, given the other clouds we have in production we could just delete all ubuntu-trusty images there and let it not build new instances of that until it gets a new upload01:56
clarkbfungi: I imagine it owuld help a lot if the disk image was half the size01:57
fungiyeah, that too01:57
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Revert "Use swapfile environment in CI"
clarkbosic images are building now01:58
clarkbwhere building == uploading01:58
*** kzaitsev_mb has quit IRC01:58
clarkbjenkins02 is still going :|01:59
*** tiswanso has joined #openstack-infra01:59
*** tiswanso has quit IRC01:59
*** tiswanso has joined #openstack-infra02:00
cody-somervillejeblair: any thoughts on underlining corresponding letter to keybinding in buttons?
nibalizercody-somerville: i like that02:02
clarkbI just kicked off another ubuntu trusty build for zlib02:04
*** HenryG has quit IRC02:06
fungiclarkb: i often resort to putting jenkins02 into shutdown mode to get it to update faster/with fewer api errors02:06
*** HenryG has joined #openstack-infra02:07
clarkb2/4 images uploaded to osic02:08
clarkbno slaves yet02:08
*** superdan is now known as dansmith02:09
*** apoorvad has quit IRC02:12
clarkbquick someone generate some load02:14
fungineed me to burn out a ton of ubuntu-trusty ready nodes? i can pretty trivially with gertty mass reviews02:18
clarkbfungi: yes please :)02:18
clarkbI mostly just want to boot a VM there and demand is too low with the ratios02:19
clarkball 4 images are in osic now02:19
clarkbthe dib image build for ubuntu-trusty to add zlib is in progress02:19
clarkb(we will want ot queue up reuploads for that image once it is done02:19
clarkband jenkinses are jjbing02:20
fungii went lightly, but should be quite a few launching in moments02:20
clarkbyup there is one now thank you02:20
clarkbthe trick here is going to be did we get the floating ip pool stuff set properly (I believe so)02:21
*** asselin__ has joined #openstack-infra02:21
clarkbjenkins02 is into xstatic02:21
*** jcooley_ has quit IRC02:21
clarkbI think it may have 502'd earlier but instead of dealing with it immediately I think we do 04 and 05 then come back to 0202:22
clarkbsince 02 is mostly up to date and slow as molasses02:22
clarkband its ready \o/02:22
fungii would shutdown 02, clean up and restart it once it's idle, and update jjb with it offline02:22
* clarkb does a happy dance and waits for a job to start on it02:22
clarkbready means that fips and all that work02:22
fungi02 has a tendency to get pretty awful (worse than 01 for some reason)02:23
clarkbnote that the osic flavor is 4 vcpu instead of 8, but testing tempest we still run super fast02:24
clarkband that job succeeded02:24
*** apoorvad has joined #openstack-infra02:24
clarkband the node is deleted02:25
clarkbwe have successfully clouded02:25
*** _ari_ has joined #openstack-infra02:26
clarkb02 just stopped updating I am going to fg 0402:26
clarkband leave 02 for later02:26
*** _ari_ is now known as ari02:29
*** ari is now known as _ari_02:29
clarkbfungi: that job failed due to bad config in glance but it also said there was no mysql db02:32
clarkbfungi: which would be a regression on the unittests if that is consisten02:32
*** _ari_ has quit IRC02:33
cody-somervilleWow. I just realized the primary developer/maintainer of urwid lives here in Ottawa.02:34
fungiclarkb: indeed, the experimental coverage job may not set up the databases. i should check the voting version02:35
*** |-paul-| has quit IRC02:36
fungiclarkb: hrm, they both call the mysql-prep and mysql-db macros02:36
fungiclarkb: and you can see the sudo mysql commands happening in the console log on that job too02:37
*** baoli has joined #openstack-infra02:38
fungiclarkb: are you talking about the "NOTICE:  database "openstack_citest" does not exist, skipping DROP DATABASE" message (note that it's wrapped to the next line)?02:39
*** _ari_ has joined #openstack-infra02:39
fungiclarkb: the failure looks like it's a discover issue... glance's normal coverage job may have always been broken02:39
fungiNon-zero exit code (1) from test listing.02:39
fungiwhich i think may be precipitated by the line above that02:40
fungiUnrecognized option '[report] ignore-errors=' in config file .coveragerc02:40
*** sbalukoff has quit IRC02:41
funginot to mention this is coverage for glance on stable/kilo02:41
fungiso it's also probable nobody cares that job's been broken02:42
*** thorst has quit IRC02:42
*** thorst has joined #openstack-infra02:43
fungias for the "errors" from the database stup macros, they attempt to drop the database they're going to create in case it already exists, for the sake of keeping idempotent02:43
fungiand also to make these versions of the jobs safe to run on the bare-trusty instances if we'd ended up taking that route temporarily02:44
*** _ari_ has quit IRC02:46
*** tqtran has quit IRC02:46
*** _ari_ has joined #openstack-infra02:46
*** woodster_ has quit IRC02:47
*** thorst has quit IRC02:51
*** _ari_ has quit IRC02:52
*** _ari_ has joined #openstack-infra02:53
*** _ari_ has quit IRC02:53
*** _ari_ has joined #openstack-infra02:54
*** _ari_ has quit IRC02:55
*** _ari_ has joined #openstack-infra02:55
*** _ari_ has quit IRC02:55
*** apoorvad has quit IRC02:55
*** _ari_ has joined #openstack-infra02:56
*** noorul has joined #openstack-infra02:59
*** _ari_ has quit IRC03:01
*** _ari_ has joined #openstack-infra03:01
*** bpokorny has joined #openstack-infra03:02
openstackgerritMerged openstack/requirements: Need a new minimum for oslo.cache in Mitaka
*** Jeffrey4l has joined #openstack-infra03:03
fungithose experimental bindep jobs were a carpet bombing run. we only analyzed the results of ones who failed when their non-bindep counterparts passed03:06
nibalizerclarkb: fungi would you be willing to 2a this?
fungithought i already had03:07
fungiapproved now03:07
clarkbI just kicked off some uploads now that the ubuntu-trusty image is rebuilt with zlib things03:07
fungilooks like it doesn't get applied anywhere that /etc/logstash would be created for other purposes03:07
*** noorul has quit IRC03:08
clarkbI am not doing rax because thats not super reliable nayways03:09
*** gildub has joined #openstack-infra03:09
clarkbfungi: if we are brae we can set the max servers to 100 and go weekend :)03:10
*** ddieterly has quit IRC03:14
*** baoli has quit IRC03:16
openstackgerritMerged openstack-infra/puppet-subunit2sql: Switch /etc/logstash to /etc/subunit2sql
fungiclarkb: well, the next phase is to scour project-config for any job configs still mentioning bare-trusty and update them (and figure out which are inheriting bare-trusty from projects.yaml) and then start switching all the devstack-trusty jobs over to bare-trusty03:18
clarkb04 finished but errored so I have restarted its jjb updates03:19
fungioh, you mean max-servers on osic03:19
fungii was thinking min-ready on ubuntu-trusty for some reason03:19
clarkbyes that :)03:19
clarkbbut we can do both03:19
fungiwhich we're nowhere near ready to need that many on hand03:19
fungibut yeah, osic sounds good. i'll be around all weekend03:20
clarkbI mostly want the ratio higher so that the chances of getting an ubuntu-trusty-2-node pair are higher which can be used to say the 2-node setup works well enough there03:22
fungiyeah, that would be great03:22
clarkbthen we can add devstack-trusty-2-node03:22
clarkbso far it looks good and no leaked fips03:22
clarkbalso ubuntu-trusty up to date everywhere but rax, ovh, and internap03:24
mtreinishfungi: oh, reading the backscroll about the coverage jobs, do you think it's worth adding the post pipeline to subunit2sql collection?03:26
clarkbit probably won't hurt03:27
mtreinishgranted if it's failing before there's any subunit output that means it won't show up on openstack-health, but I forgot about those jobs03:27
clarkbthough only coverage generates subunit there03:27
clarkbfungi: maybe tomorrow morning we can bump osic and make sure it stays happy?03:27
clarkbI am running out of steam and cooking dinner03:27
clarkb04 is done, starting 0503:28
clarkbI should probably figure out this grafyaml thing and add an osic03:29
nibalizerfungi: thanks!03:31
mtreinishnibalizer: hmm, I'm surprised it didn't pull in the patch:
mtreinishit ran after it merged03:32
*** mtanino has joined #openstack-infra03:32
nibalizerit can take up to 15 minutes03:33
nibalizerso next run will definitely have it03:33
mtreinishnibalizer: ah, ok03:34
openstackgerritClark Boylan proposed openstack-infra/project-config: Grafana for OSIC
*** fawadkhaliq has quit IRC03:37
fungiclarkb: sounds great03:40
fungihave a good evening03:40
clarkbinternap has up to date ubuntu-trsuty03:41
clarkbso just ovh left and they are uploading03:41
clarkbfungi: I am going to put 02 in shutdown mode now03:41
*** esikachev has joined #openstack-infra03:42
fungithat's often the only way i can complete a large update on it03:42
*** esikachev has quit IRC03:46
*** thorst has joined #openstack-infra03:49
*** HenryG has quit IRC03:54
*** thorst has quit IRC03:56
*** yamamoto has joined #openstack-infra03:58
*** yamamoto has quit IRC04:01
*** tiswanso has quit IRC04:08
*** kingia_ has joined #openstack-infra04:20
*** yamamoto has joined #openstack-infra04:21
*** HenryG has joined #openstack-infra04:22
*** jtomasek has joined #openstack-infra04:23
*** kingia has quit IRC04:23
*** Daisy has joined #openstack-infra04:31
*** yamamoto has quit IRC04:31
*** claudiub|2 has joined #openstack-infra04:34
*** dimtruck is now known as zz_dimtruck04:34
*** Daisy has quit IRC04:35
*** claudiub has quit IRC04:37
*** zz_dimtruck is now known as dimtruck04:41
*** treaki_ has joined #openstack-infra04:43
*** treaki has quit IRC04:47
*** roxanaghe has joined #openstack-infra04:51
*** dimtruck is now known as zz_dimtruck04:52
*** thorst has joined #openstack-infra04:54
*** Keedya_86 has joined #openstack-infra04:58
*** bpokorny has quit IRC04:59
*** Keedya_86 has quit IRC04:59
*** thorst has quit IRC05:01
*** keedya has quit IRC05:02
*** HenryG has quit IRC05:11
*** HenryG has joined #openstack-infra05:13
*** HenryG has quit IRC05:14
*** claudiub|2 has quit IRC05:14
*** HenryG has joined #openstack-infra05:14
*** Jeffrey4l has quit IRC05:17
*** yamamoto has joined #openstack-infra05:32
*** zz_dimtruck is now known as dimtruck05:34
*** yamamoto has quit IRC05:38
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests
*** hdd has joined #openstack-infra05:41
*** esikachev has joined #openstack-infra05:42
*** dimtruck is now known as zz_dimtruck05:44
*** esikachev has quit IRC05:47
*** salv-orlando has joined #openstack-infra05:52
*** ashleighfarnham has joined #openstack-infra05:53
*** salv-orl_ has quit IRC05:55
*** vivekd has joined #openstack-infra05:59
*** thorst has joined #openstack-infra05:59
*** thorst has quit IRC06:06
*** harlowja_at_home has quit IRC06:07
*** hdd has quit IRC06:19
clarkbok 05 is all done and I think 02 is finally happy but rerunning one more time there to see it noop06:20
clarkbI probably won't stay awake to watch it06:20
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** Daisy_ has joined #openstack-infra06:20
*** HenryG has quit IRC06:20
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** vivekd has quit IRC06:30
*** ashleighfarnham has quit IRC06:51
*** fawadkhaliq has joined #openstack-infra06:54
*** yamahata has joined #openstack-infra06:54
*** fawadkhaliq has quit IRC06:56
*** roxanaghe has quit IRC07:00
*** Daisy_ has quit IRC07:02
*** thorst has joined #openstack-infra07:05
*** vivekd has joined #openstack-infra07:05
*** thorst has quit IRC07:11
*** zz_dimtruck is now known as dimtruck07:13
*** Daisy has joined #openstack-infra07:23
*** Daisy has quit IRC07:27
*** vivekd has quit IRC07:40
*** esikachev has joined #openstack-infra07:43
*** esikachev has quit IRC07:47
*** sputnik13 has quit IRC07:49
*** roxanaghe has joined #openstack-infra08:01
*** roxanaghe has quit IRC08:06
*** thorst has joined #openstack-infra08:09
*** thorst has quit IRC08:16
openstackgerritAndreas Jaeger proposed openstack/requirements: Fix other-requirements
AJaegerinfra-root, dhellmann, ttx: Requirements repo is broken, I expect that the change above will fix it. Please merge it to unblock requirements repo08:19
*** HeOS has quit IRC08:19
AJaegerinfra-root, on I got:08:20
AJaeger"Unable to connect to"08:20
AJaegerIs ovh broken?08:20
AJaegerinfra-root, ovh-bhs1 seems to have broken networking, I see lots of image failures there08:26
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable ovh-bhs1
AJaegerinfra-root, a band-aid suggestion: disable ovh-bhs1 ^08:28
*** [1]Thelo has joined #openstack-infra08:29
*** asselin__ has quit IRC08:30
openstackgerritAndreas Jaeger proposed openstack/requirements: Fix other-requirements
*** Thelo has quit IRC08:32
*** [1]Thelo is now known as Thelo08:32
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert storlets, trove, zuul jobs to ubuntu-trusty
*** achanda has quit IRC08:36
*** achanda has joined #openstack-infra08:40
*** esikachev has joined #openstack-infra08:42
*** esikachev has quit IRC08:46
*** achanda has quit IRC08:50
openstackgerritAndreas Jaeger proposed openstack/requirements: Fix other-requirements
*** sbalukoff has joined #openstack-infra09:00
*** thorst has joined #openstack-infra09:14
*** thorst has quit IRC09:20
*** dimtruck is now known as zz_dimtruck09:22
*** HeOS has joined #openstack-infra09:24
*** roxanaghe has joined #openstack-infra09:30
*** roxanaghe has quit IRC09:34
*** rguillebert has joined #openstack-infra10:05
*** e0ne has joined #openstack-infra10:05
*** yamahata has quit IRC10:06
*** e0ne has quit IRC10:08
*** yamamoto has joined #openstack-infra10:09
*** vivekd has joined #openstack-infra10:11
*** yamamoto has quit IRC10:13
*** thorst has joined #openstack-infra10:18
*** thorst has quit IRC10:26
*** esikachev has joined #openstack-infra10:27
*** yamamoto has joined #openstack-infra10:28
*** yamamoto has quit IRC10:31
*** yamamoto has joined #openstack-infra10:35
*** yamamoto has quit IRC10:35
*** vivekd has quit IRC10:42
*** HeOS has quit IRC10:45
*** [HeOS] has joined #openstack-infra10:45
*** esikachev has quit IRC10:49
*** claudiub|2 has joined #openstack-infra10:53
*** [HeOS] has quit IRC10:54
*** HeOS has joined #openstack-infra10:56
*** aspiers has quit IRC10:56
*** aspiers has joined #openstack-infra10:57
*** HeOS has quit IRC11:03
*** HeOS has joined #openstack-infra11:06
*** ogelbukh has quit IRC11:07
*** claudiub|2 has quit IRC11:15
*** roxanaghe has joined #openstack-infra11:18
*** roxanaghe has quit IRC11:23
*** thorst has joined #openstack-infra11:23
*** thorst has quit IRC11:30
*** pcaruana has quit IRC11:34
*** yamamoto has joined #openstack-infra11:36
*** yamamoto has quit IRC11:43
*** Jeffrey4l has joined #openstack-infra11:47
*** ddieterly has joined #openstack-infra11:49
*** salv-orl_ has joined #openstack-infra11:52
*** salv-orlando has quit IRC11:55
*** Daisy has joined #openstack-infra11:57
*** claudiub|2 has joined #openstack-infra12:01
*** HeOS has quit IRC12:03
*** dims has joined #openstack-infra12:11
*** dims has quit IRC12:13
*** lhinds has joined #openstack-infra12:14
lhindshi, would openstack-infra be a good place to ask about an issue with the wiki (OpenID) issue?12:14
*** ddieterly is now known as ddieterly[away]12:17
*** ddieterly[away] is now known as ddieterly12:17
*** dkehn has quit IRC12:19
*** gildub has quit IRC12:23
*** thorst has joined #openstack-infra12:28
*** thorst has quit IRC12:36
*** ddieterly has quit IRC12:49
AJaegerlhinds: in general yes - but not right now. Best come back Monday after 15:00 UTC12:54
*** zeih has joined #openstack-infra12:56
*** thorst has joined #openstack-infra12:57
*** thorst has quit IRC13:02
*** thorst has joined #openstack-infra13:03
*** roxanaghe has joined #openstack-infra13:06
*** ihrachys has joined #openstack-infra13:07
openstackgerritAndreas Jaeger proposed openstack/requirements: Fix other-requirements
*** dims has joined #openstack-infra13:08
dimsAJaeger : hey13:10
dimsAJaeger : looks like is a rush?13:10
openstackgerritMerged openstack-infra/project-config: Sort bindep-fallback.txt
AJaegerdims: yes. Either we remove bindep-fallback or fix it ;)13:11
*** thorst has quit IRC13:11
*** roxanaghe has quit IRC13:11
AJaegerdims, we switched from bare-trusty - which ignored other-requirements - to ubuntu-trusty which uses it13:11
AJaeger(I meant other-requirements above not bindep-fallback)13:11
AJaegerthanks, dims. Hope it now includes all dependencies...13:12
AJaegerinfra-root, seems that ovs-bhs1 is ok again...13:13
dimsthanks for working through it AJaeger13:13
AJaegeryou're welcome13:14
*** yamamoto has joined #openstack-infra13:15
*** yamamoto has quit IRC13:15
*** yamamoto has joined #openstack-infra13:15
*** openstackgerrit_ has quit IRC13:17
openstackgerritAndreas Jaeger proposed openstack/requirements: Fix other-requirements
AJaegerdims: Just updated again - saw pypy failing as well...13:17
dimsAJaeger : no worries13:18
*** openstackgerrit_ has joined #openstack-infra13:18
AJaegerlet's see whether it passes.13:18
*** Daisy has quit IRC13:18
* AJaeger will be back later13:18
dimsAJaeger : y will watch it13:21
*** zeih has quit IRC13:23
*** rfolco has joined #openstack-infra13:23
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Add heat-translator>=0.4.0
pabelangerAJaeger: Hows bindep jobs looking this morning? I think clarkb and fungi got the latest fixes merged but haven't researched any failures in the gate13:27
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Bump python-neutronclient to 4.1.1
*** ddieterly has joined #openstack-infra13:27
*** rfolco has quit IRC13:29
*** amotoki has joined #openstack-infra13:30
*** ddieterly has quit IRC13:31
*** daemontool_ has quit IRC13:32
openstackgerritMerged openstack-infra/project-config: Clean up bindep-fallback.txt
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** hdd has joined #openstack-infra13:37
openstackgerritPaul Belanger proposed openstack-infra/project-config: Remove experimental bindep infra jobs
*** yamamoto has quit IRC13:43
openstackgerritFrancesco Longo proposed openstack-infra/project-config: Added IoTronic project.
*** dims has quit IRC13:53
AJaegerpabelanger: requirements repo had a broken other-requirements file, hope it's fixed now.13:54
AJaegerpabelanger: otherwise no bindep related problems that showed up13:54
AJaegerpabelanger: - that one needs to be enhanced for CentOS at one point...13:55
pabelangerAJaeger: Ya, we'll need to get centos-7 dib going soon, and come up with a way for projects to validate bindep on centos (even fedora)13:58
AJaegerpabelanger: remember to test on requirements repo - or better: On each repo that has other-requirements file13:58
*** hdd has quit IRC13:59
AJaegerpabelanger: if you want to review some more moves to ubuntu-trusty, please check
AJaegerpabelanger: but if people just recheck without checking what's broken (some requirements jobs had 3+ rechecks) and fixing it or asking how to fix it, we will not learn about it ;(14:02
pabelangerAJaeger: indeed. I've been scanning status.o.o/zuul trying to catch failures14:03
AJaegerthis morning ovh-bhs1 was completely broken, every job failed there since the mirror couldn't be reached ;(14:04
AJaegerThat "helped" a lot in finding and fixing issues ;(14:04
*** chlong_ has quit IRC14:05
*** hdd has joined #openstack-infra14:08
*** thorst has joined #openstack-infra14:08
*** hdd has quit IRC14:09
*** mtanino has quit IRC14:13
*** claudiub has joined #openstack-infra14:14
*** HenryG has joined #openstack-infra14:15
*** claudiub|2 has quit IRC14:15
*** nihilifer has quit IRC14:15
*** thorst has quit IRC14:15
*** nihilifer has joined #openstack-infra14:17
*** Daisy has joined #openstack-infra14:19
*** openstackgerrit_ has quit IRC14:20
*** gnuoy_ has joined #openstack-infra14:21
*** openstackgerrit_ has joined #openstack-infra14:21
*** roxanaghe has joined #openstack-infra14:22
*** Daisy has quit IRC14:23
*** dkehn has joined #openstack-infra14:25
*** gnuoy_ has quit IRC14:26
*** asselin__ has joined #openstack-infra14:27
*** roxanaghe has quit IRC14:27
*** asselin__ has quit IRC14:31
*** openstackgerrit_ has quit IRC14:32
*** openstackgerrit_ has joined #openstack-infra14:33
*** yamamoto has joined #openstack-infra14:44
*** dims has joined #openstack-infra14:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: ubuntu-trusty: misc,openstack-planet,...
*** sdake has joined #openstack-infra14:51
*** yamamoto has quit IRC14:52
*** sdake_ has joined #openstack-infra14:56
*** sdake has quit IRC14:57
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert devstac* to ubuntu-trusty
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert elastic-recheck to ubuntu-trusty
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert grenade to use ubuntu-trusty
fungilhinds: i can imagine there's one significant issue you'll encounter with openid authentication to the wiki. if you're trying to create a new account (don't have an existing account on the wiki or switching to a different account which didn't exist before) then that won't work just now. we've got new account creation disabled while we work through a serious spam problem there15:12
AJaegerfungi, thanks for approving 288858 - should merge in 4 mins since dims was faster ;)15:13
openstackgerritMerged openstack/requirements: Fix other-requirements
AJaegerthat was 4 seconds instead of 4 mins ;)15:13
*** hdd has joined #openstack-infra15:13
*** thorst has joined #openstack-infra15:13
fungiAJaeger: yeah, i missed the other approval on it when i skimmed, saw it after i approved15:14
fungigood catch15:14
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert job-builder to use ubuntu-trusty
*** dizquierdo has joined #openstack-infra15:15
fungiAJaeger: also looks like some requirements changes in the gate passed behind that, so should be good15:15
AJaegerfungi, pabelanger and myself have some changes for ubuntu-trusty conversion and cleanup up - nothing urgent. is complete list15:15
AJaegerfungi: yes, requirements looks fine. That other-requirements file was never tested ;(15:16
fungifor some reason i thought someone had tested out all the repos with other-requirements.txt files in them but now i don't recall who (and know it wasn't me)15:16
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Do not merge - TESTING
fungiand also your database package additions there remind me that i'll want to start thinking about maybe making the database setup job macros no-op when the executables they call aren't in path15:18
AJaegerI think we had added jobs for it but might we forgot them somehow ;(15:18
AJaegerfungi: good idea...15:18
fungithat would allow projects who don't need database support for any of their project-specific jobs to share the same job templates as those who do, but easily omit them from other-requirements.txt15:19
* AJaeger tests now the repos with other-requirements ...15:19
fungithe down-side to making that autodiscovered is that the database tests in many jobs are also opportunistic, so a bug preventing installation of database tools could go silently undetected and those projects would just stop running their opportunistic database tests15:20
*** thorst has quit IRC15:21
fungianother option would be to have a databases option to the install-distro-packages macro which appends the required packages to the other-requirements.txt before feedint it to bindep, but that way leaves a hole where a project not using the databases can still quietly grow a dependency on a package which one of the database packages depends on15:21
openstackgerritMerged openstack/requirements: Add heat-translator>=0.4.0
AJaegerfor now I would rather get rid of firefox and other odd bits from bindep-fallback15:23
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: DO NOT MERGE - just testing
*** sdake_ is now known as sdake15:24
fungiyep, longer term i'd love for projects which need special database setup to run some of their tests start doing it explicitly (maybe moving the tooling to do the setup into their repo if we can standardize on what that script would be named so it can be consistently called from jobs)15:25
fungibut also the fallback list does provide us with opportunities for some low-hanging fruit, like getting horizon an other-projects.txt tested out so we can drop web browsers from the fallback list ;)15:25
fungiup-side to this design is that projects wanting to add a replacement package list are able to self-test that15:26
AJaegerwhat about branches? our fallback list is branch independent, so horizon needs to add other-projects for stable branches as well, doesn't it?15:27
fungialso wanting to augment devstack to use bindep rather than its current home-grown package lists (maybe teh devstack requirements can even move into individual repos as well that way)15:27
fungiAJaeger: if we're going to remove an entry from the fallback list, yes it needs to first be set up for all branches of whatever was consuming that15:27
fungibut again, the per-project other-requirements.txt additions are self-testing, just need to remember to add them to all branches which need one15:28
openstackgerritMerged openstack/requirements: Bump python-neutronclient to 4.1.1
AJaegerfungi, so next we should document and announce some of this. What do you think of writing an announcement email to openstack-dev?15:29
fungiit's removals from the fallback list which aren't tested, because figuring out what might break and automatically running jobs for that is a mostly intractible problem15:29
* AJaeger saw some requirements job rechecked 3+ times that failed because of ubuntu-trusty...15:29
fungiAJaeger: yeah, i planned to write up an announcement once the bulk of common jobs are off bare-trusty and not broken, so probably monday15:29
AJaegerfungi, thanks15:29
AJaegerfungi, do we have documentation for the bindep work? If not, where should it live?15:30
fungiAJaeger: rechecks from failures because of the broken package name in the requirements repo's other-requirements.txt file, or for other failures on ubuntu-trusty?15:30
AJaegerbecause of broken package names in requirements file - so they either never looked at log files or didn't understand failure15:31
fungiAJaeger: we have documentation published for bindep, but could stand to have some simple details somewhere (i haven't thought about where) to document our testing implementation which is using it15:31
AJaegerbut might have been that one job failure was for the broken ovh-bhs cloud which self-healed15:31
fungiand yeah, sorry i wasn't on the computer before that fixed itself. i'm curious to know what went wrong in ovh15:32
fungimaybe they were performing network maintenance or something15:32
AJaegerfungi: no worries - it was soo idle that I just documented it here ;)15:32
fungias in you've already written some documentation?15:33
fungior are you talking about ovh?15:33
AJaegerfungi: I meant: Documented the ovh-bhs failure on the irc channel15:33
fungiokay, cool. i need to go run some errands for a couple hours, but will give the announcement and test documentation some thought and see what i come up with15:33
fungiand then will be around the rest of the day in case there's more we need to test/fix too15:33
fungithanks for keeping an eye on it on your saturday!15:34
AJaegerfungi, just one question: I was thinking of shutting down ovh-bhs with lowering quote on it - but had to self-approve my own change.15:34
AJaegerIF there would have been real load, I would have been tempted - is that kind of self-approval in such situations fine or shouldn't I think of that?15:35
fungiAJaeger: in situations like this, self-approval is fine if there's nobody else around and you're just dropping capacity temporarily. especially at low-volume times15:35
AJaegerit was so low-volume that it wasn't needed ;) Ok, thanks - and we can always revert;)15:36
funginot involving others when there's someone around to help is a bit of a problem, but if you're stuck by yourself, i'd rather people feel empowered to fix what they can and we can work out later whether it was the right call given the circumstances15:37
AJaegerargh, bindep repo is broken with the other-requirements file ;(15:37
fungioh wow15:37
fungiat least that's fixable15:37
AJaegeryep, looking at it...15:37
fungii bet it doesn't have any database packages15:38
AJaegerfungi, you won the bet ;)15:38
fungiwhich we've basically introduced as a requirement since we're using the same job-template regardless of whether a project has database-oriented tests15:38
fungiso, yeah, probably any project with an other-requirements.txt file and using our typical tox job-templates is likely to need those added15:39
fungii probably want to think up a solid plan for solving that soon, because when we announce bindep integration in our jobs the flood gates will open on projects adding other-requirements.txt files and i'd rather not tell them that they have to add database packages even if they don't have any database-using tests15:40
AJaegerfungi, I agree - so for our initial documentation, let's discourage other-requirements15:40
*** dims has quit IRC15:41
fungimaybe i'll have a grand solution in a couple hours when i get back15:41
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Fix other-requirements
*** dims has joined #openstack-infra15:43
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Fix other-requirements
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Fix other-requirements
dhellmannAJaeger, fungi : where can I go to read more about bindep? I have no idea what it is. It's not showing up in a search of infra specs.15:46
dhellmannAJaeger : thanks15:47
AJaegerdhellmann: but major change here is that with going from bare-trusty to ubuntu-trusty we use a diskimage-builder to setup the images15:48
AJaegerAdvantage is that we now can run ubuntu-trusty on all clouds while bare-trusty was rackspace only15:48
*** _nadya_ has joined #openstack-infra15:48
AJaegerWe install packages now at gate setup. There's a default package list in project-config/jenkins/data/bindep-fallback.txt - and projects can use their own list called other-requirments.txt. See my changes above for bindep and release-tools..15:49
dhellmannok, I had heard of it but didn't know anything about it so was a bit alarmed to start seeing comments like "all the requirements jobs are failing" or whatever and wanted to learn a bit more so I could participate in fixing things (assuming you don't have it worked out by monday)15:49
AJaegerdhellmann: the fix was
dhellmannAJaeger : how do I make sure that a job like the release announce job, that uses release-tools but isn't triggered by a commit to that repo, has the right dependencies installed?15:50
dhellmannfor now the release part of the tools is still being run locally, but that will have a similar requirement. merging something to openstack/releases will eventually run a job that installs release-tools and then runs some of the scripts15:52
AJaegerdhellmann: best discuss with fungi about this15:52
dhellmannAJaeger : ok, it'll wait until next week then15:52
* AJaeger checks that job now15:53
AJaegerdhellmann: Plan B would be to do something like install-distro-packages macro does and add your own package list for this case - or like install-distro-packages-fallback-test, hardcoding a list. See jenkins/jobs/macros.yaml15:55
*** Qiming has quit IRC15:55
dhellmannAJaeger : ok, building it into the job makes sense15:55
AJaegerdhellmann: thanks for raising this. I'm sure there are some more situations where we need to extend the current setup...15:56
AJaegerfungi, please read scrollback later ^15:56
*** ddieterly has joined #openstack-infra15:57
* AJaeger will be back later...15:57
*** thorst has joined #openstack-infra16:01
*** dims has quit IRC16:04
*** ddieterly has quit IRC16:05
*** thorst has quit IRC16:08
*** thorst has joined #openstack-infra16:09
*** max_lobur has quit IRC16:10
*** ddieterly has joined #openstack-infra16:11
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow filtering of worklists and boards by story or task
*** dims has joined #openstack-infra16:16
*** thorst has quit IRC16:17
*** dims has quit IRC16:18
*** Jeffrey4l has quit IRC16:19
*** ddieterly is now known as ddieterly[away]16:23
*** pradk has quit IRC16:38
*** max_lobur has joined #openstack-infra16:43
*** asselin__ has joined #openstack-infra16:46
*** thorst has joined #openstack-infra16:52
*** armax has quit IRC17:00
*** armax has joined #openstack-infra17:01
*** ddieterly[away] is now known as ddieterly17:06
*** fedexo has joined #openstack-infra17:07
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Only create 1024MB swap in tripleo-ci
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
*** sdake_ has joined #openstack-infra17:17
openstackgerritSpencer Krum proposed openstack-infra/puppet-askbot: Add captcha to the askbot venv
*** asselin has joined #openstack-infra17:17
*** sdake has quit IRC17:18
*** ddieterly is now known as ddieterly[away]17:20
*** asselin__ has quit IRC17:20
*** ddieterly[away] has quit IRC17:23
*** jamesmcarthur has joined #openstack-infra17:25
*** sdake has joined #openstack-infra17:27
*** sdake_ has quit IRC17:29
*** sdake_ has joined #openstack-infra17:31
*** sarob has joined #openstack-infra17:32
*** ddieterly has joined #openstack-infra17:34
*** sdake has quit IRC17:34
*** sarob has quit IRC17:36
*** ihrachys has quit IRC17:40
*** ihrachys has joined #openstack-infra17:41
*** ihrachys has quit IRC17:45
*** e0ne has joined #openstack-infra17:45
*** esikachev has joined #openstack-infra17:46
*** armax has quit IRC17:47
*** max_lobur has quit IRC17:49
*** esikachev has quit IRC17:50
*** salv-orlando has joined #openstack-infra17:52
fungidhellmann: AJaeger: short story is this is supposed to be transparent for projects without a other-requirements.txt file in them17:52
fungithe problem with the requirements repo was that someone had added that file to it17:53
fungiand it wasn't tested, so was broken17:53
nibalizermtreinish: started up the subunit worker on subunit0217:53
*** jamesmcarthur has quit IRC17:54
fungidhellmann: AJaeger: so when the jobs changed to support people adding that file later (something we don't want them doing prematurely until we're sure this is a good path forward and doesn't need further tweaking), it insta-broke requirements jobs that tried to use that file17:54
dhellmannfungi : ah, so someone jumped the gun adding that file to the requirements repo?17:54
fungidhellmann: more or less17:55
*** salv-orl_ has quit IRC17:55
fungii think it was added for someone's local convenience, or the thought that it might be convenient for local use for some devs, but never actually tested to confirm it was correct17:55
fungidhellmann: another fix would have been for us to just delete that file from the repo17:55
*** _nadya_ has quit IRC17:56
fungithe goal was for this transition not to require any changes to any existing projects, that impact was an oversight17:56
fungiout of the ~1k projects in gerrit, something like 4 have apparently added an other-requirements.txt file17:57
*** mriedem has joined #openstack-infra17:57
fungiand the requirements repo was one of them17:57
dhellmannok, cool17:57
dhellmannI'll read the docs about the tool this week so I know more about it17:58
fungidhellmann: making a big announcement has the potential to backfire, since the announcement would be "don't use this yet" and projects are almost certain to decide to start doing it then17:58
dhellmannoh, of course :-)17:58
*** ddieterly is now known as ddieterly[away]17:58
fungiwe still have some rough edges around in-job database setup which we've worked around for now but will make the other-requirements.txt files unnecessarily list things which those projects don't actually need, so i want to finish addressing that piece before we declare open season17:59
fungialso, some deep soul searching this morning has brought me to the conclusion that "opportunistic testing" is fragile and counter to proper ci, so i think i'm going to start a -dev ml thread to discuss ways we can resolve that18:00
fungiwith regards to things like nova unit tests going "oh, there's no database for me to use, i'll just skip these tests then"18:01
*** jamesmcarthur has joined #openstack-infra18:01
fungii'd love to come up with a good answer for devs to be able to run whatever tests their local environments support while still making it a hard failure in the ci if the needs of a particular set of tests aren't met18:01
clarkbfungi: should we go ahead and bump the osic max-servers number now? I can write that patch if you want to do it18:02
fungiclarkb: sounds like a great time, yes18:02
AJaegerclarkb, fungi: Could you review the two other broken other-requirements files, please? for bindep18:02
AJaeger for release-tools - dhellmann, feel free to +2 as well ;)18:03
AJaegerclarkb: go for it...18:03
*** harlowja_at_home has joined #openstack-infra18:03
AJaegerOh, we need to backport the requirements change to all branches ;(18:04
AJaegerLet me do that as well...18:04
fungiAJaeger: i'm starting to worry the other-requirements.txt files in this handful of repos is going to set a bad example or cause it to be blindly copied around. maybe better to just propose deletion of them for now?18:04
openstackgerritClark Boylan proposed openstack-infra/project-config: Use entire osic quota
fungithen we can add them back later once we work out the database solution so that we're not claiming they all use mysql and postgres for tests when few (if any) actually do \18:05
AJaegerfungi: I was wondering that as well - ok, let me propose removing them...18:05
clarkbAJaeger: fungi on we use mysql-python and not PyMySQL?18:05
AJaegerclarkb: see project-config/jenkins/data/bindep*18:06
clarkbalso I am not sure that is the right package for psycopg218:06
clarkboh! it is because we use the client directly in the jobs18:06
AJaegerclarkb: then the bindep-fallback list is wrong18:06
AJaegerclarkb: another reason to delete the file? ;)18:06
fungiAJaeger: well, not wrong18:06
*** ddieterly[away] is now known as ddieterly18:07
fungiclarkb: to support the handful of projects (like nova, ironic) which expect mysql and postgres to be set up with test databases so that they don't skip some tests, the job-templates need the databases installed to be able to set them up18:07
fungii doubt that libmysqlclient-dev is actually required there18:07
openstackgerritAndreas Jaeger proposed openstack/requirements: Remove other-requirements.txt
fungimacros in some of the job templates are using the command-line db clients to set up database permissions to support the projects which use them, mimicking how we preconfigure the databases on bare-trusty images18:08
clarkbthose dev headers are used by mysql-python or is python-mysql to link against the C lib18:08
fungiclarkb: i think neither18:09
fungiclarkb: i think libmysqlclient-dev is just being copied into that list needlessly18:09
clarkblibpq-dev is the one that psycopg2 links against18:09
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Remove other-requirements.txt
fungiclarkb: in 288904 those entries are being added purely so that when the job macro to set database permissions runs (even though none of bindep's tests are ever going to use those databases) the job doesn't fail18:10
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Remove other-requirements.txt
fungiwhich is why i'm more in favor of just deleting stray other-requirements.txt files from projects in the short term18:11
clarkbor +2 as it goes18:11
fungiuntil we can work out how to make projects not need to declare false dependencies on database services and clients purely because it was easy to reuse the same job-templates for now18:11
clarkbfungi: I think the macro to install and configure the dbs can vendor their own package installs18:12
clarkbthen if individual projects need headers to use libs against them they declair those in other requirements?18:12
fungimy only remaining worry with deleting it from the requirements repo is that one of its jobs may be running on devstack-trusty and calling bindep to decide what packages to install, rather than being there purely for convenience (though if that were the case, i'd have expected the invalid entry in the file to be causing problems for that job). anyway the changes to delete the file should tell us18:13
fungipretty quickly18:13
*** ddieterly is now known as ddieterly[away]18:13
fungiclarkb: my concern there is that projects may silently grow dependencies on packages used by those macros, either directly or indirectly18:14
fungiwhat i'm starting to think we ultimately need is a version of those job-templates which use databases to run their tests, and another version for the masses18:14
clarkban alternative woud be to have the projects declare the deps and install first, then if mysql/postgres/whatever is present configure it18:14
fungibut that means figuring out which projects' tests are actually using the databases we preconfigure on bare-trusty18:15
AJaegerfungi, I created a tox-nodb macro already for individual tox invocations18:15
AJaegerand use that for manuals18:15
AJaegerbut yes, we need to figure this out in each case ;(18:15
fungiAJaeger: yep, in the end we probably want that inverted so that the normal macros are the ones without extra database setup18:15
*** keedya_ has quit IRC18:16
*** keedya has joined #openstack-infra18:16
fungiand just the handful of projects which use preconfigured databases for some tests use the special template which does that part at the start of the job (after package installs but before revoking sudo)18:16
AJaegerfungi, just a global rename ;)18:16
*** Sukhdev has joined #openstack-infra18:16
AJaegerfungi, we can add a tox-db macro - or however it's called, move over and rename18:17
*** jamesmcarthur has quit IRC18:17
clarkbnibalizer: fungi AJaeger should be ready18:17
clarkbas is
*** thorst has quit IRC18:18
AJaegerdhellmann: requirements changes are and - could you help pushing them through, please18:18
fungiAJaeger: if only it were that simple. the issue is that the "safe" position is to have everyone use the version with the databases preconfigured until we can determine which ones don't need it, and that seems like it's going to be tough to figure out in light of the "opportunism" in a lot of our database tests18:18
* AJaeger trusts clarkb and hands out a +218:18
*** thorst has joined #openstack-infra18:18
AJaegerfungi, yes I know - have a look at .18:19
fungibecause things like nova unit tests will run perfectly fine without database setup today, but unless you actually compare test counts or look in the list of tests they run, you won't know they're missing some18:19
AJaegerI hope I updated so far correctly for the non-db case in these.18:19
nibalizerclarkb: ooo!18:19
*** e0ne has quit IRC18:19
*** harlowja_at_home has quit IRC18:20
fungiwhat i want is a solution to make tests like that _not_ opportunistic, at least when we run ci jobs18:20
fungithough that also means devs and the gate running tests in slightly different ways18:20
clarkbfungi: maybe the thing to do is put the db setup in tox itself?18:20
clarkbthen it fails if it can't do its thing and project notices18:21
clarkbthis requires devs to have a mysql to point nova at but that isn't too hard18:21
fungiclarkb: i also thought about that, but i'm not keen on the idea of opening projects up to running arbitrary shell as root in their gate jobs18:22
fungii worry that will turn into the swiss-army-knife of terrible solutions for them18:22
fungiat least the bindep step running as root is consistently running a set of preinstalled tools against a declarative data file, not just running whatever a dev decided to tack onto the tox.ini in a special as-root env18:23
fungisome jobs (e.g. functional and integration testing) it's inevitable, but i fear projects getting to the point where we're making world-writeable directories because their unit tests need to write to arbitrary file locations on the system or something18:24
fungitoo much flexibility in those jobs would be detrimental18:25
fungiAJaeger: i want to see if those requirements changes pass, mainly because i worry that something like the requirements integration test script might have actually been using that file18:26
fungiin which case this gets slightly uglier18:26
lifelessthose would not be unit tests18:26
fungilifeless: more or less my point, yeah18:27
*** thorst has quit IRC18:27
lifelessthey also wouldn't be well isolated18:27
lifelesswhich is IMO a bigger issue than a name18:27
fungiright, the "unit" tests in nova and ironic which do opportunistic database interactions now are arguably also not really unit tests18:27
AJaegerSo far coverage, python27, python34, pypy are setup for db access - others not. We could create two set of templates one with db, one without and let projects migrate...18:27
*** sdake_ is now known as sdake18:27
fungiAJaeger: yeah, that is what i'm assuming we're probably going to have to do18:28
AJaegerbindep (change 288904) and release-tools (change 288905) passed the check18:29
*** [1]Thelo has joined #openstack-infra18:29
AJaegerlifeless: I think you added other-requirements to some of the 4 repos, do you want to review the removal that I'm currently doing? lists these and a few other changes...18:30
*** sputnik13 has joined #openstack-infra18:31
fungiAJaeger: so this coming week i'll get changes up to switch the projects in question over to versions of unit test job-templates which don't perform database setup, and then we can add other-requirements.txt files back to them (the readditions will also be self-testing, which will be helpful)18:31
*** Thelo has quit IRC18:31
lifelessAJaeger: removal? why?18:31
*** [1]Thelo is now known as Thelo18:31
AJaegerlifeless: see backscroll18:32
AJaegerlifeless: we discussed this 40 mins or so ago18:32
lifelessAJaeger: sunday morning before a plane trip... unless its urgent, I'll read tomorrow18:32
AJaegerlifeless: read tomorrow ;)18:33
lifelessAJaeger: the only other-requirements I added, from memory, was pbr and requirements, and their scripts use it, so it should fail gate tests if you try to remove it18:33
fungilifeless: it's non-urgent. temporary removals of files not used in existing jobs (assuming the test for them pass) which will be added back safely in a self-testing manner in a few days18:33
AJaegerlifeless: we can revert the removal once infra is ready - right now, they cause some problems...18:33
openstackgerritMerged openstack-infra/project-config: Use entire osic quota
*** e0ne has joined #openstack-infra18:34
openstackgerritMerged openstack-infra/bindep: Remove other-requirements.txt
AJaegerlifeless: safe travels!18:34
fungiAJaeger: lifeless: if some jobs for those repos are actually using those files (which i suspected they might) then we'll just have to correct them in-place and add a few things so that they're sufficient for also running all the python jobs too, until we can clean that up18:35
*** armax has joined #openstack-infra18:35
AJaegerfungi, indeed requirements repo uses the other-requirements file.18:36
AJaegermy tests are failing18:36
lifelessAJaeger: thanks; just to syd for the bug smash18:36
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: IPv4 network isolation testing for HA and Ceph
lifelessfungi: not sure what cleanup is needed, but yeah - today is bad.18:37
fungilifeless: right, we were trying to do it today to avoid unforeseen impact while everyone was submitting patches during the week18:37
fungiokay, so we'll have to go back to your original plan of adding all the other packages necessary to support the job-templates instantiated on it, and maybe also a big comment around those stating that they're temporary additions, and also a comment in the file reminding people not to go copying that pattern around until we're better able to support it in all our jobs18:38
fungiAJaeger: ^18:38
openstackgerritMerged openstack-infra/elastic-recheck: Remove argparse from requirements
AJaegerfungi, we're only discussing requirements repo, correct?18:38
fungiAJaeger: right (or any others where the temporary removal fails because they had some custom job which used bindep to parse that file in their repo)18:39
fungia future to-do was to come up with a good recommended pattern for per-job other-requirements.txt entries using specially-named tags18:40
AJaegerok, next iteration for requirements ;)18:40
clarkbnibalizer: can haz grafana too?
AJaegerper-job other-requirements is indeed another optimization - but I'm not up for the drawing board today18:40
nibalizeryep thats yaml18:41
fungibut right now the other-requirements.txt file will have to include all distro packages for any job on that project incorporating the install-distro-packages macro18:41
clarkbnibalizer: that is basically what I said as I replaced vexxhost with osic18:41
*** e0ne has quit IRC18:43
AJaegerfungi, the requirements repo only has an ubuntu version, should I try updating it for CentOS as well?18:43
openstackgerritSpencer Krum proposed openstack-infra/system-config: Send ansible-puppet timing information to graphite
*** dizquierdo has quit IRC18:45
openstackgerritAndreas Jaeger proposed openstack/requirements: Comment other-requirements.txt
*** e0ne has joined #openstack-infra18:46
fungiAJaeger: nah, i wouldn't bother adding more than necessary to that file for now18:48
openstackgerritAndreas Jaeger proposed openstack/requirements: Comment other-requirements.txt
AJaegergood ;)18:48
fungijust the minimum entries to get the packages and database macros to run successfully18:48
*** hdd has quit IRC18:49
lifelessfungi: put the temporary requirements in a dedicated profile ?18:51
*** ddieterly[away] is now known as ddieterly18:51
lifelessfungi: that is what they are there for....18:51
AJaegerfungi, clarkb, dhellmann: is the backport of requirements change to stable/liberty so that it works in our gates. Please review and approve18:52
*** armax has quit IRC18:52
openstackgerritMerged openstack-infra/project-config: Grafana for OSIC
*** ddieterly has quit IRC18:52
AJaegerclarkb: woot!18:53
clarkb is another18:55
*** sputnik13 has quit IRC18:55
*** e0ne has quit IRC18:57
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list
*** e0ne has joined #openstack-infra19:01
fungilifeless: i eventually want the install-distro-packages job macro to support job-specific bindep profiles, but the initial simple design relies on the list to contain the set of all packages needed by all jobs which use that macro19:01
fungiin the default profile19:02
fungiAJaeger: 288928 looks like master to me19:04
AJaegerfungi: it's failing at - and I'm confused19:05
AJaegeryes, something looks strange there ;(19:06
*** sdake_ has joined #openstack-infra19:06
AJaegerlooking at it looks fine - but looking at a version using git review -d ...;git show it looks broken ;(19:07
AJaegeram I mixing up numbers?19:08
* AJaeger double checks19:08
*** sdake has quit IRC19:08
AJaeger288928 is master 288929 is stable/liberty19:09
*** sdake has joined #openstack-infra19:10
fungiAJaeger: the 288929 commit looks sane to me. i wonder if there was already something broken with stable/liberty requirements unit tests?19:11
AJaegerfungi, I leave this to the requirements experts19:12
fungiyeah, something is causing it to fail several of its negative parser checks19:12
*** sdake_ has quit IRC19:12
AJaegerlifeless, dhellmann : could you check 288929 tomorrow, please? I don't understand why it suddenly fails - and we need this backport to unbreak stable/liberty requirements19:12
fungipkg_resources._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "':///foo#'"19:13
fungiexcept, that seems like what it _should_ be doing for the openstack_requirements.tests.test_requirement.TestToReqs.test_not_urls test19:13
fungiso why is that a test failure?19:14
AJaegerfungi, no idea - and I'm signing off now...19:15
AJaegerfungi, have a great saturday!19:16
fungithanks for working on this AJaeger and have a good evening/weekend!19:16
AJaegerthanks, fungi19:16
*** e0ne has quit IRC19:18
*** thorst has joined #openstack-infra19:18
fungilooks like it's been a couple weeks since that job ran on stable/liberty so it's possible something has broken how the exceptions are being interpreted in those specific tests19:20
*** ddieterly has joined #openstack-infra19:21
fungii'm going back through the git history on requirements master now to see if there was a fix merged for something like this in the past couple weeks that we might need to backport and squash into19:22
*** e0ne has joined #openstack-infra19:22
fungitaking a while. feature freeze seems to have been a popular time to get requirements changes crammed in19:24
fungiaha, "Tolerate both new and old setuptools"19:25
fungii'll get that backported to stable/liberty and squashed together with the other-requirements.txt update19:25
*** ihrachys has joined #openstack-infra19:26
*** e0ne has quit IRC19:28
clarkbfungi: ok I sent mail to osic letting them know we are live with links to a successful tempest job run and the grafana dashboard19:29
*** thorst has quit IRC19:29
clarkbtempest full took 37 minutes so that looks good19:30
*** ihrachys has quit IRC19:30
*** e0ne has joined #openstack-infra19:32
fungidhellmann: mriedem: so stable/liberty requirements unit tests were broken for a week or so already. latest patchset of combines a backport of AJaeger's fix from today for our recent job changes with a backport of dims's patch from earlier for dealing with new setuptools19:33
*** achanda has joined #openstack-infra19:34
*** achanda has quit IRC19:34
fungiclarkb: excellent news--thanks!!!19:34
*** e0ne has quit IRC19:35
*** e0ne has joined #openstack-infra19:38
clarkboh I forgot to mention the thing about specifying the cinder volume attach point19:40
mriedemfungi: looking19:40
*** e0ne has quit IRC19:41
mriedemfungi: AJaeger: question inline19:43
*** e0ne has joined #openstack-infra19:46
clarkbok updated email sent with volume attach details19:46
*** e0ne has quit IRC19:48
*** Sukhdev has quit IRC19:50
*** Sukhdev has joined #openstack-infra19:50
*** e0ne has joined #openstack-infra19:52
*** claudiub|2 has joined #openstack-infra19:53
*** claudiub has quit IRC19:56
*** e0ne has quit IRC19:58
*** zul has quit IRC20:01
*** e0ne has joined #openstack-infra20:03
*** e0ne has quit IRC20:06
*** _nadya_ has joined #openstack-infra20:09
*** e0ne has joined #openstack-infra20:11
*** ddieterly is now known as ddieterly[away]20:11
dhellmannAJaeger , fungi : thanks for digging into that. I'll wait for the answer to mriedem's question before approving20:14
*** zul has joined #openstack-infra20:15
fungimriedem: i think AJaeger's previous patchset was actually squashing two changes from master already but didn't indicate that in the commit message20:18
fungihe's gone for the day but i'll see if i can fix up the commit message to better reflect that20:19
AJaegermriedem: see, that's the version taht is backported20:20
AJaegerfungi, just came back to clarify this ;)20:20
AJaegerfungi, yes, indeed my commit message was not clear - if you want to update this, highly appreciated.20:21
fungii will do so--thanks AJaeger and enjoy your weekend!20:22
AJaegerthanks a lot, fungi!20:22
*** ihrachys has joined #openstack-infra20:22
*** eharney has quit IRC20:23
*** max_lobur has joined #openstack-infra20:25
*** e0ne has quit IRC20:27
*** ihrachys has quit IRC20:28
*** _nadya_ has quit IRC20:30
*** e0ne has joined #openstack-infra20:30
*** e0ne has quit IRC20:33
*** e0ne has joined #openstack-infra20:37
*** thorst has joined #openstack-infra20:38
*** keedya has quit IRC20:38
*** claudiub|2 has quit IRC20:39
*** e0ne has quit IRC20:40
openstackgerritMerged openstack/requirements: Comment other-requirements.txt
*** thorst has quit IRC20:43
*** Sukhdev has quit IRC20:54
fungimriedem: dhellmann: i've answered the questions on and updated the commit message accordingly20:58
*** e0ne has joined #openstack-infra20:58
fungilikely not urgent as nobody'd noticed for a while that branch has been broken by newer setuptools, and there are no open patches aside from that one anyway21:01
openstackgerritMerged openstack-infra/devstack-gate: Add support for ipmitool driver in Ironic
*** ddieterly[away] has quit IRC21:11
*** e0ne has quit IRC21:11
*** fawadkhaliq has joined #openstack-infra21:12
*** jtomasek has quit IRC21:20
*** dims has joined #openstack-infra21:20
*** jtomasek has joined #openstack-infra21:23
*** Sukhdev has joined #openstack-infra21:23
*** Daisy has joined #openstack-infra21:23
*** angdraug has joined #openstack-infra21:24
*** |-paul-| has joined #openstack-infra21:24
*** Daisy has quit IRC21:28
*** dabukalam_ is now known as dabukalam21:30
*** ihrachys has joined #openstack-infra21:40
*** jtomasek has quit IRC21:41
*** thorst has joined #openstack-infra21:41
*** ihrachys has quit IRC21:44
*** thorst has quit IRC21:48
*** thorst has joined #openstack-infra21:48
*** salv-orlando has quit IRC21:52
*** salv-orlando has joined #openstack-infra21:52
*** Sukhdev has quit IRC21:53
*** HeOS has joined #openstack-infra22:04
*** ddieterly has joined #openstack-infra22:22
*** Jeffrey4l has joined #openstack-infra22:25
*** dkehn has quit IRC22:25
*** fawadkhaliq has quit IRC22:27
*** skraynev has quit IRC22:28
*** kambiz has quit IRC22:31
openstackgerritPaul Belanger proposed openstack-infra/bindep: Do not merge!
*** skraynev has joined #openstack-infra22:31
*** trey has quit IRC22:32
*** Clint has quit IRC22:32
*** dkehn has joined #openstack-infra22:33
*** kuromagi has quit IRC22:33
*** snikitin has quit IRC22:33
*** cburgess has quit IRC22:34
*** waynr has quit IRC22:34
*** pleia2 has quit IRC22:35
*** kambiz has joined #openstack-infra22:36
*** snikitin has joined #openstack-infra22:36
*** pleia2 has joined #openstack-infra22:37
*** kuromagi has joined #openstack-infra22:38
*** sdake has quit IRC22:38
*** Clint has joined #openstack-infra22:39
*** Clint has quit IRC22:39
*** Clint has joined #openstack-infra22:39
*** salv-orlando has quit IRC22:39
*** cburgess has joined #openstack-infra22:40
*** waynr has joined #openstack-infra22:40
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a quote button to comments
*** trey has joined #openstack-infra22:45
*** ddieterly is now known as ddieterly[away]22:45
*** snikitin has quit IRC22:45
*** thorst has quit IRC22:46
*** thorst has joined #openstack-infra22:47
*** Jeffrey4l has quit IRC22:48
*** |-paul-| has quit IRC22:49
*** ddieterly[away] is now known as ddieterly22:50
*** snikitin has joined #openstack-infra22:50
*** ddieterly is now known as ddieterly[away]22:52
*** thorst has quit IRC22:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow filtering of worklists and boards by story or task
*** nijaba_ has quit IRC22:57
*** amotoki has quit IRC22:58
*** nijaba has joined #openstack-infra23:03
*** nijaba has quit IRC23:03
*** nijaba has joined #openstack-infra23:03
*** sdake has joined #openstack-infra23:13
*** dims has quit IRC23:16
*** jamesmcarthur has joined #openstack-infra23:17
*** sdake has quit IRC23:21
*** ihrachys has joined #openstack-infra23:28
*** ihrachys has quit IRC23:32
*** fedexo has quit IRC23:35
*** salv-orlando has joined #openstack-infra23:40
*** baoli has joined #openstack-infra23:44
*** aspiers has quit IRC23:46
*** salv-orlando has quit IRC23:47
*** fawadkhaliq has joined #openstack-infra23:48
*** ddieterly[away] is now known as ddieterly23:49
*** baoli has quit IRC23:52
*** thorst has joined #openstack-infra23:53

Generated by 2.14.0 by Marius Gedminas - find it at!