Luca's meaningless thoughts   SponsorGitHub SponsorsLiberapayPaypalBuy Me A CoffeePatreonFlattr

Calling abort() on unhandled exception

by Leandro Lucarella on 2010- 07- 30 15:01 (updated on 2010- 07- 30 15:01)
tagged abort, d, en, exception, mail, rant - with 0 comment(s)

This is one of those threads where I want to drop my work on the GC and kill some of the D's main developers.

Please, at least learn a little about abort() before giving an opinion!

Presenting CDGC

by Leandro Lucarella on 2010- 07- 28 21:48 (updated on 2010- 07- 28 21:48)
tagged cdgc, d, dgc, en, gc, git, intro, repository - with 0 comment(s)

I've just published the git repository of my D GC implementation: CDGC. The name stands for Concurrent D Garbage Collector but right now you may call it Configurable D Garbage Collector, as there is no concurrency at all yet, but the GC is configurable via environment variables :)

It's based on the Tango (0.99.9) basic GC, there are only few changes at the moment, probably the bigger ones are:

  • Runtime configurability using environment variables.
  • Logging of malloc()s and collections to easily get stats about time and space consumed by the GC (option malloc_stats_file [str] and collect_stats_file [str]).
  • Precise heap scanning based on the patches published in bug 3463 (option conservative [bool]).
  • Runtime configurable debug features (option mem_stomp [bool] and sentinel [bool]).
  • Other non user-visible cleanups.

The configuration is done via the D_GC_OPTS environment variable, and the format is:

D_GC_OPTS=opt1=value:opt2=value:bool_opt:opt3=value

Where opt1, opt2, opt3 and bool_opt are option names and value is their respective values. Boolean options can omit the value (which means true) or use a value of 0 or 1 to express false and true respectively. String options have no limitations, except they can't have the : char in their values and they have a maximum value length (255 at this moment).

At the moment is a little slower than the Tango basic GC, because the precise scanning is done very naively and a lot of calls to findPool() are done. This will change in the future.

There is a lot of work to be done (cleanup, optimization and the concurrent part :), but I'm making it public because maybe someone could want to adapt some of the ideas or follow the development.

C++ template WTF

by Leandro Lucarella on 2010- 07- 25 23:22 (updated on 2010- 07- 25 23:22)
tagged c++, d, en, programming, template, wtf - with 0 comment(s)

See this small program:

template<typename T1>
struct A {
        template<typename T2>
        void foo_A() {}
};

template<typename T>
struct B : A<T> {
        void foo_B() {
                this->foo_A<int>(); // line 10
        }
};

int main() {
        B<int> b;
        b.foo_B();
        return 0;
}

You may think it should compile. Well, it doesn't:

g++ t.cpp -o t
t.cpp: In member function ‘void B<T>::foo_B()’:
t.cpp:10: error: expected primary-expression before ‘int’
t.cpp:10: error: expected ‘;’ before ‘int’

Today I've learned a new (horrible) feature of C++, foo_A is an ambiguous symbol for C++. I've seen the typename keyword being used to disambiguate types before (specially when using iterators) but never a template. Here is the code that works:

template<typename T1>
struct A {
        template<typename T2>
        void foo_A() {}
};

template<typename T>
struct B : A<T> {
        void foo_B() {
                this->template foo_A<int>();
                //    ^^^^^^^^
                // or: A<T>::template foo_A<int>();
                // but not simply: template foo_A<int>();
        }
};

int main() {
        B<int> b;
        b.foo_B();
        return 0;
}

Note how you have to help the compiler, explicitly saying yes, believe me, foo_A is a template because it has no clue. Also note that the template keyword is only needed when A, B and A::foo_A are all templates; remove the template<...> to any of them, and the original example will compile flawlessly, so this is a special special special case.

Yeah, really spooky!

In D things are more natural, because templates are not ambiguous (thanks to the odd symbol!(Type) syntax), you can just write:

class A(T1) {
        void foo_A(T2)() {}
}

class B(T) : A!(T) {
        void foo_B() {
                foo_A!(int)();
        }
}

void main() {
        B!(int) b;
        b.foo_B();
}

And all works as expected.

Proposition Infinity

by Leandro Lucarella on 2010- 07- 24 23:10 (updated on 2010- 07- 24 23:10)
tagged en, futurama, gay, homosexual, humor, marriage, proposition infinity, tv - with 0 comment(s)

I'm sorry if I'm being a little repetitive about Futurama lately, but I just saw Proposition Infinity (6ACV04) which talks about egalitarian marriage between humans and robots (obviously parodying gay marriage legalization).

Well, it turns out, in case you didn't know, in Argentina, egalitarian marriage (AKA gay marriage) is legal since July 15th. This Futurama episode was aired on the July 8th, so it was kind of premonitory (leaving out the fact that the law in Argentina was in discussion long before that :).

Anyway, cheers to all homosexual couples that can get married now in Argentina, and to all robosexual couples that will be able to get married sometime after the year 3000 =P.

The attack of the killer app

by Leandro Lucarella on 2010- 07- 22 21:08 (updated on 2010- 07- 22 21:08)
tagged en, futurama, humor, the attack of the killer app, tv, video, youtube - with 0 comment(s)

The attack of the killer app (6ACV03) is one of the best and geekest episodes of Futurama I've ever seen.

Making fun of iPhone and Twitter is really easy, and done everywhere now days, but Futurama does it in another level, in a level only geeks can enjoy that much. One clear example (leaving aside the title of the episode) is this little jewel:

Fry: Since when is the Internet about robbing people of their privacy?

Bender: August 6th, 1991.

Referencing the announcement of the creation of the WorldWideWeb.

Rocanrrrrrollllll n-n-n!!!

by Leandro Lucarella on 2010- 07- 21 00:02 (updated on 2010- 07- 21 00:02)
tagged asado violento, canon, es, loudquietloud, music, pixies, self - with 0 comment(s)

Hoy estuve de tour por el centro y aproveché para comprar entradas para dos bandas legendarias que se juntan para rockear. Aunque una levemente más popular que la otra, 8 veces más popular si la popularidad fuera proporcional al precio de la entrada, pero creo que el precio favorece a la banda menos conocida :P

Se trata de Pixies (apurensennn que mañana es el último día con descuento, las dos módicas gambas que sale un campo son con descuento!) y Asado Violento (apurensennnn que tocan el viernes).

Les dejaría una foto de las entradas para que se mueran de envidia, pero la razón por la cual salí a pasear en primera instancia es que mi hermosa camarita anda con problemas así que la llevé a la garantía (al negocio porque las importan ellos y Canon Argentina se lava las manos, y Canon USA también). Tengo un poco de miedo con la garantía por esto y por esto, espero que sean excepciones, aunque ya de por sí me dijeron que llame en 20 o 30 días :S. Igual debo darles un par de porotos porque fueron muy amables e hicieron tudu bom, tudu legal (me dieron comprobante con fecha, con los datos de la cámara y todo lo que les dejaba). Como buen justiciero que soy, los mantendré informados al respecto.

Por esta razón lamentablemente no habrá reporte gráfico del primer recital, espero que para el segundo ya esté todo listo.

¡Ah! También vi quietLOUDquiet, está buena, aunque me dejó una sensación amarga ver que los geniales Pixies son un rejunte de patéticos perdedores =P. Creo que Joey Santiago es el único más o menos normal (y tal vez Frank Black si no lo hubieran filmado yéndose a dormir escuchando el cassette de auto-ayuda).

Zuiikin' English

by Leandro Lucarella on 2010- 07- 18 00:20 (updated on 2010- 07- 18 00:20)
tagged en, english, humor, japanese, video, wtf, youtube, zuiikin - with 0 comment(s)

I spent hours laughing at this Zuiikin' English videos.

Debian unstable + amd64 + flashplugin-nonfree

by Leandro Lucarella on 2010- 07- 17 23:25 (updated on 2010- 07- 17 23:25)
tagged adobe, amd64, debian, en, flash, plugin, rant - with 0 comment(s)

I've done my regular aptitude full-upgrade and noticed the f*cking Adobe Flash player stopped working for amd64.

If you are experiencing the same, take a look at the workaround here.

Flash must die, DIE!

Futurama S06

by Leandro Lucarella on 2010- 07- 16 20:54 (updated on 2010- 07- 16 20:54)
tagged comedy central, en, futurama, s06, tv - with 0 comment(s)

Today is a happy day. Even when the new Futurama season started in Comedy Central some weeks ago, I completely forgot about it and I didn't see any new episode until today.

I really missed the sensation of viewing a Futurama episode that I didn't see several times before (there were some movies, but it's not the same).

Oh! What a beautiful series! Really good first episode, very well spent 20 minutes.

Performance WTF

by Leandro Lucarella on 2010- 07- 14 03:47 (updated on 2010- 07- 25 03:11)
tagged d, dgc, en, gc, make, memory layout, performance, voronoi, wtf - with 0 comment(s)

How do I start describing this problem? Let's try to do it in chronological order...

Introduction

I've collected a bunch of little programs to use as a benchmark suite for the garbage collector for my thesis. I was running only a few manually each time I've made a change to the GC to see how things were going (I didn't want to make changes that degrade the performance). A little tired of this (and missing the point of having several tests using just a few), I've decided to build a Makefile to compile the programs, run the tests and generate some graphs with the timings to compare the performance against the current D GC (Tango really).

The Problem

When done, I noticed a particular test that was notably slower in my implementation (it went from ~3 seconds to ~5 seconds). Here is the result (see the voronoi test, if you can read the labels, there is some overlapping because my effort to improve the graph was truncated by this issue :).

https://llucax.com/blog/posts/2010/07/13-time.png

But I didn't recall it being that way when running the test manually. So I ran the test manually again, and it took ~3 seconds, not ~5. So I started to dig where the difference came from. You'll be surprised by my findings, the difference came from executing the tests inside the Makefile!

Yes, take a look at this (please note that I've removed all output from the voronoi program, the only change I've made):

$ /usr/bin/time -f%e ./voronoi -n 30000
3.10
$ echo 'all:' > Makefile
$ echo -e '\t$C' >> Makefile
$ make C="/usr/bin/time -f%e ./voronoi -n 30000"
/usr/bin/time -f%e ./voronoi -n 30000
5.11
$

This is not just one isolated run, I've tried hundreds of runs and the results are reproducible and stable.

Further Investigation

I don't remember exactly how I started, but early enough, noticing that the Tango's basic GC didn't suffered from that problem, and being my GC based on that one, I bisected my repository to see what was introducing such behaviour. The offending patch was removing the difference between committed and uncommitted pages in pools. I can see that this patch could do more harm than good now (I didn't tried the benchmark when I did that change I think), because more pages are looped when working with pools, but I can't see how this would affect only the program when it's executed by Make!!!

I had a patch that made thing really nasty but not a clue why they were nasty. I've tried everything. First, the obvious: use nice and ionice (just in case) to see if I was just being unlucky with the system load (very unlikely since I did hundreds of runs in different moments, but still). No change.

I've tried running it on another box. Mine is a Quad-Core, so I've tried the Dual-Core from work and I had the same problem, only the timing difference were a little smaller (about ~4.4 seconds), so I thought it might be something to do to with the multi-cores, so I've tried it in a single core, but the problem was the same (~10.5 seconds inside make, ~7 outside). I've tried with taskset in the multi-core boxes too. I've tried putting all the CPUs with the performance governor using cpufreq-set too, but didn't help.

Since I'm using DMD, which works only in 32 bits for now, and since my box, and the box at work are both 64 bits, I suspected from that too, but the old AMD is 32 bits and I see the problem there too.

I've tried valgrind + callgrind + kcachegrind but it seems like valgrind emulation is not affected by whatever difference is when the program is ran inside make because the results for the run inside and outside make were almost identical.

I've tried env -i, just in case some weird environment variable was making the difference, but nothing.

I've tried strace too, to see if I spotted anything weird, and I saw a couple of weird things (like the addresses returned by mmap being suspiciously very different), but nothing too concrete (but I think inspecting the strace results more thoughtfully might be one of the most fertile paths to follow). I took a look at the timings of the syscalls and there was nothing taking too much time, most of the time is spent in the programs calculations.

So I'm really lost here. I still have no idea where the difference could come from, and I guess I'll have to run the tests from a separate shell script instead of directly inside make because of this. I'll ask to the make developers about this, my only guess is that maybe make is doing some trickery with the scheduler of something like that for the -j option. And I'll take a look to the offending patch too, to see if the performance was really degraded and maybe I'll revert it if it does, no matter what happen with this issue.

If you have any ideas on what could be going on, anything, please let me know (in a comment of via e-mail). Thanks :)

Update

I've posted this to the Make mailing list, but unfortunately didn't got any useful answer. Thanks anyway to all the people that replied with nice suggestions!

Update

Thanks Alb for the investigation, that was a 1/4kg of ice-cream well earned =P

A couple of notes about his findings. An easy way to trigger this behaviour is using the command setarch, the option -L changes the memory layout to ADDR_COMPAT_LAYOUT, see the commit that introduced the new layout for more details.

The call to setrlimit(RLIMIT_STACK,  RLIM_INFINITY) by Make (which has a reason) triggers that behaviour too because the new layout can't have an unlimited stack, so using ulimit (ulimit -s unlimited) causes the same behaviour.

The same way, if you type ulimit -s 8192 ./voronoi as a command in a Makefile, the effect is reverted and the command behaves as outside the Makefile.

Part of the mystery is solved, but a question remains: why the test is so address-space-layout dependant? It smells like a GC bug (present in the basic GC too, as other tests I've done show the same odd behaviour, less visibly, but still, probably because of the removal of the distinction between committed and uncommitted memory patch).

Update

Last update, I promise! =)

I think I know what is adding the extra variance when the memory layout is randomized: false pointers.

Since the GC is conservative, data is usually misinterpreted as pointers. It seems that are address spaces that makes much more likely that simple data is misinterpreted as a valid pointer, at least for the voronoi test. This is consistent with other tests. Tests with random data notably increases their variance among runs and are pretty stable when the memory layout is not randomized.

I'll try to give the patch to integrate precise heap scanning a try, and see if it improves things.

What remains a mystery is what happened with the committed memory distinction, now I can't reproduce the results. I made so many measures and changes, that maybe I just got lost in a bad measure (for example, with the CPU using the ondemand governor). I've tried again the tests with and without that change and the results are pretty the same (a little better for the case with the distinction, but a really tiny difference indeed).

Well, that's all for now, I'll give this post a rest =)

Update

Don't believe me, ever! =P

I just wanted to say that's is confirmed, the high variance in the timings when heap randomization is used is because of false pointers. See this comment for more details.

Blu's Muto

by Leandro Lucarella on 2010- 07- 05 00:54 (updated on 2010- 07- 05 03:30)
tagged animation, blu, buenos aires, graffiti, muto, stop motion - with 0 comment(s)

Here is Muto (an ambiguous animation painted on public walls), a stop motion animation by the street artist Blu:

You can also download it in HD (it's CC licensed) via Bittorrent.

As a side note, parts of this film has been done in Buenos Aires, particularly very close to my actual and previous homes. There are a quite a few of other Blu paintings in both neighborhoods, some can be viewed in his home page:

I'd always wanted to take some pictures to all that nice graffiti, but never did. Maybe I should.

By the way, I've end up in this video via a top 10 stop motion videos in youtube. Not all videos are great, but some really are, I recommend viewing at least number 1, 2 (Muto :), 3, 4, 7, 9 and 10.

Enjoy!

Update

Blu just released a new video (with another scene in Buenos Aires, I can't tell where, but I can recognize the trash bins and sidewalks =):

Alemanes

by Leandro Lucarella on 2010- 07- 03 22:41 (updated on 2010- 07- 03 22:41)
tagged 2010, alemania, argentina, es, fútbol, humor, mundial, sudáfrica, video, youtube - with 2 comment(s)

Empezamos así:

Y terminamos así:

Que se le va a hacer... A chuparla, a seguiarla mamando hasta el 2014.

Pixies @ Argentina

by Leandro Lucarella on 2010- 07- 02 12:38 (updated on 2010- 07- 02 12:38)
tagged argentina, buenos aires, en, live, luna park, music, pixies - with 0 comment(s)

From http://www.pixiesmusic.com/:

Pixies are Coming to Argentina!

Wednesday, October 6, 2010
Buenos Aires, Argentina
Luna Park
ticketing information to be announced shortly....

Whoa!

Bacap

by Leandro Lucarella on 2010- 07- 02 01:02 (updated on 2010- 07- 02 01:02)
tagged bacap, backup, en, floss, release, rsync, software - with 0 comment(s)

I've published my backup script (and named it Bacap, which is how it sounds when you pronounce "backup" in Spanish ;), as it suddenly became used by somebody else and received some patches =)

It has a simple home page for those wanting to see what is it about. I won't be doing formal releases thought, since is such a simple script, that would be overkill (just be thankful it has a git repo and a home page ;).