Archive for category Coding
Jinja2 Bytecode Cache
Posted by Carlos Daniel in Coding on July 21, 2012
I have been a bit busy and thus off from turbogears development for a while, but I see 2.2 release shaping up really good, props for Michael and Alessandro for their work.
Now, looking at the stuff that I’m most familiar with (jinja2 support code), I have been thinking for a while that jinja2 does not have bytecode caching enable by default, in fact, there is no support at all in tg2 code for that, while Genshi does have caching available jinja2 is lacking in this area, where enabling the cache is actually very simple.
I created a few tests with the latest 2.2rc2 turbogears with jinja bytecode caching enabled, I used the included filesystem based cache and a custom (simple) in-memory cache, as we can see on the graphics from the benchmark the gains from enabling the cache a bit small, around 4 request per second more, but if you are looking to squeeze performance out of tg2 this could be a good option as its practically free, the cache is not really intrusive, if I make a change to the template it is reloaded automatically thus it doesn’t affect the normal development workflow.
Jinja2 Bytecode Benchmark: Cached vs Uncached
I should, however, do some additional test under gunicorn or uwsgi to see if there are even more substantial benefits to enabling the bytecode cache.
Turbogears 2.2 Beta: Benchmarking
Posted by Carlos Daniel in Coding on April 11, 2012
TG 2.2 beta has been released and with it many improvements, I was quite curious about the results of the performance enhancements derived of moving TG to Crank, I do not expect to see a big improvement as this is only a Beta and there is still many things to do and tweak before final release, also the real performance work will come from the 2.3 branch, but for now I would like to post my benchmark results.
I’m using the same benchmarking method as I used to tests the templating systems, this is now easier as 2.2 integrates many jinja improvements, the difference this time is that I will be benchmarking the build-in Paster server vs Gunicorn vs UWSGI, lets see how it goes.
Paster
As we can see templating performance status quo is more or less the same, Genshi is still slow but there has been good improvements, the environ page rendering is now as fast as other templates, but overall is Genshi in 2nd and Jinja and Mako more or less tied in 1st place.
Overall performance has improved in comparison with 2.1 as you can see, ~20 rps more has been squeezed in this release.
Gunicorn
gunicorn_paster 0.14.2 was used in this test, running with 2 sync workers.
With Gunicorn we can see that the Genshi performance on the environ rendering test is no fluke, here Jinja is only a bit faster than Mako, overall speed wise 2.2 Beta is very similar to 2.1 under Gunicorn but Mako is now faster (as fast as Jinja) and Genshi has got a nice speedup too.
uWSGI
uwsgi 1.1.2 was used for this test, with 2 workers, with Master option (-M), using –http :8080 option to serve directly instead of using uwsgi protocol (to serve under nginx).
Finally the uWSGI server, here we can see TG 2.2 shine, uWSGI is a bit faster overall than Gunicorn, templating performance is similar in this test, Jinja and Mako are more or less tied.
Closing
As we can see Turbogears 2 is pretty much alive and kicking, development is going to ramp up pretty soon as tw2 sprints and migration has keeped the TG team busy but that is mostly done, it would be also awesome if people could give TG 2.2 Beta a try and report problems, to do so install as per TG Instructions but use the beta package repository instead of the regular one:
easy_install -i http://tg.gy/beta tg.devtools
Developing on OSX
Posted by Carlos Daniel in Coding on March 29, 2012
I have always seen a lot of people speak highly of Mac OSX as a development platform and Apple hardware as a high quality platform in general, lately I have been in charge of some mobile development, a small OpenGL app, which I initially wrote on my linux machine taking care of making it very cross platform, then it came the time to port to IOS, the port was pretty simple and most things worked out of the box without much trouble.
As much as it sounds like a story with a happy ending I got to say it was a short but hellish walk when porting, stuff mostly worked ok, there where things like OpenAL that did not work as expected and required a bit of extra work, but the one thing that drove me mad was the general OSX interface, not the graphical interface per see, but the details that where totally different from what I was used to, like keybindings, for example, when editing code, I expect the End key to go to the end of the line, on OSX it just scrolls at the end of the file without moving the cursor, stuff like that really trowed me out of balance and maked developed way slower than it should and very painful, while I do not doubt you can get accustomed to OSX but coming from other OSes where stuff work a way (which by the way works exactly the same even on Windows and other OSes) and then having it all flipped it just pisses me of and lowers my productivity.
Personally I would not have an Apple machine as my main development workstation or laptop, I’m already too accustomed to Linux and its quirks, OSX is a whole new set of quirks to workout on a platform which is less flexible to me for a relatively little gain.
Turbogears Template Benchmark
Posted by Carlos Daniel in Coding on March 21, 2012
I have been doing some work on benchmarking the many template systems under turbogears, currently the most visible are Genshi (default), Mako and now Jinja (which will be better integrated from 2.2 onwards), thus I made a simple benchmark to test the difference on speed with each template.
First I’m using TurboGears2 from git branch 215 plus my jinja tg2devtools branch, this in order to preview the jinja based quickstart template. For each templating system a fresh project is created with all defaults, each project is setup and run under paster and gunicorn for each test (this to compare development environment and a more production like environment).
The tests where done using apache bench (ab2) v2.3, with concurrency of 2 and 1000 tests, I tested 4 pages, the welcome (index), the environ page (where it renders a table from environment variables), the manager_permission_only page and the editor_user_only page, last 2 ones to test the template+sqlalchemy.
Finally those tests are not absolute and only represent performance on my machine but should give you a good idea of the performance differences between templating systems, the test machine is a AMD A6-3410MX CPU with 4 cores at 1.6ghz, 4GB RAM and 500gb 5400rpm hard disk, running under opensuse 12.1 with custom Linux kernel 3.3.0
Result data was processed with python using matplotlib to do the graphs.
Paster tests
As we can see here both mako and jinja outperform genshi by a good margin, in this test jinja is slightly faster. What is interesting however is that the welcome page Genshi is almost as fast a jinja, I tough this was a glitch or something but it works out consistently after several tests, seems to me that because the welcome page is relatively static genshi does a good job caching.
Gunicorn tests
Gunicorn tests are done by running gunicorn_paste, with 2 sync workers. The results:
Here we see it again, on the welcome page Genshi performs surprisingly on par with Jinja and Mako, but other tests really show, here Jinja is consistently faster than Mako.
Conclusions
As we can see Jinja is a good option for those looking for extra performance and are familiar already with Django templates, otherwise Mako is another great performance alternative under tg2.
Finally, I tried this test under pypy to see if I could squeeze even a bit more performance but the results where not consistent, the jit performance varies a lot, sometimes I get over 1000 request per second after jit warmup, but overall the results keep fluctuating, I can’t really say that running under pypy is a win for TG, although results seem to imply that it is a bit faster most of the time.
TurboGears2 Jinja quickstart template
Posted by Carlos Daniel in Coding on March 21, 2012
I have currently submitted a proposal for adding a set of jinja based templates to the tg2 quickstart command, with it the paster quickstart command will offer you the option to use jinja based templates, which means a full set of template files for jinja based development.
I tried to mimic as best as I could the way the Genshi templates are done, meaning that the rendering is exactly the same and that works more or less the same way, for example, using macros to define certain reusable parts of the page.
The probable timeframe for this patch to make it in mainline should be around v2.2 which is the current development branch, this patch however can work with older releases, you should have at least TG v2.1.5 (currently unreleased, under the 215 branch) for it to work as only on this version a patch for jinja template name resolution based on dotted notation has been integrated, the last means that from 2.1.5 onwards jinja will be able to load templates named after their import, such as “myproject.templates.index”, instead of the path based ones such as “index.html”.
Patches live under my jinja branch of tg2devtools for those who would like to test it earlier:
https://sourceforge.net/u/clsdaniel/tg2devtools/ci/2dd496a588d8705802e46d226a750a40bce691a4/tree/
Jinja2 on Turbogears2
Posted by Carlos Daniel in Coding on December 22, 2011
While Turbogears2 is a full stack python web framework and would seem to be opinionated, it has quite flexibility with certain components, for example, for database access there is SQLAlchemy as the default but you can also use MongoDB via Ming, which is supported in the mainline code. Template language is another thing that TG can swap, there is Genshi as default, Mako, Chameleon, Jinja, etc. Not about every template under the sun but quite a big selection for all tastes.
However, not all template languages have the same level of support, Genshi is the main one and the best supported as it is the successor of Kid (the template language on TG1), Genshi is mostly an XML/HTML driven language, it is designer friendly as it just extends XML tags it does not add any new or special syntax to XML/HTML, however, it can turn to be a bit slow and by the way Genshi is made isn’t very compatible with other Python implementations such as pypy, jython, etc. Some may opt for another templating languages to overcome those difficulties, for them there is Mako which is a text based template language, it is fast and very well supported, after all Mako was sort of the default under pylons (which TG2 uses for controller dispatch and utilities).
There are other alternatives to speedup Genshi like Chameleon and finally there is Jinja, which apparently does not fit anywhere in the equation other than add choice, however, Jinja is very similar to Django templates which should make Django developers a bit more comfortable with working with Turbogears. Jinja support was minimally there, just support for loading and rendering jinja templates, there was no support for extensions or filters (even build-in filters).
In order to bring Jinja into shape I started playing around the turbogears core code and more or less learned my way in, finally started contributing patches to add features here and there for Jinja, extensions and filters where added on 2.1.3 and on 2.1.4 build-in filters where loaded in by default, also autoloading of project defined filters was added (see tg2 Jinja docs), yesterday I was working on getting dotted template name notation working for jinja templates, so you can load templates by its package hierarchy instead of the file path (“myproject.templates.index” instead of “index.html”), which is the default modus operandis of the rest of the templates, the code is fully tested ready to be checked and merged anytime.
All those new features while nice still aren’t enough, the next step is to add Jinja templates to the base tg2 quickstart project template, so you can jumpstart a new project with jinja templates in place, which is the major hurdle for using Jinja under TG2, I’m currently working on that on a branch of the tg2devtools package, well see how it pans out.
Programming in C
Posted by Carlos Daniel in Coding on November 7, 2011
Currently I don’t program in C as much as I used to, mainly because I try to choose the right tool for the job and the fact that when I do program in C I end spending much of the time fine tuning the code (to me is like a craft). I don’t like optimizations that alter the readability of the code that much or other “underhanded tricks”, but I do care about memory usage / performance trade-offs, memory leaks, etc.
Last weekend while configuring a mail server, I remembered a small pet project I had a while ago to build a mail server, it was made in C++ and had a few interesting bits of code here and there, I still remember benchmarking it, getting around 20 to 48 messages per second.
Finally I decided to do an experiment and build a simple smtp server in pure C, using redis to manage a queue between it and a mail drop server which parses the email and drops it in the correct Maildir. In the end I did some benchmarking, getting up to 54 messages per second, not bad given that I gave up some optimizations such as using a modified tree structure to parse smtp commands.
Another further optimization I did was cutting down the amount of write calls, for clarity sake the SMTP wrote back in 2 calls, one sending the status, another the terminator character (CRLF), moving both to a single call did make an immense difference, now the benchmark is around 1100 messages per second (using 10 threads, 1kb message payload), or rather having 2 write calls slowed all the process considerably, it meant 2 syscalls, 2 roundtrips, potentially slowed the client parser too.
I could probably get even further improvements by implementing the SMTP PIPELINING extension. The maildrop server on a single thread could keep up with the smtp server so I could even squeeze a bit more performance from the whole mail pipeline.
In the end the experiment was meant to produce very lean, basic (but functional) and high performing servers, while having fun! I was very careful with memory management and does not seem to leak any, valgrind still identifies some small leaks related to hiredis client and pthreads. The memory used by the smtp server with 10 threads stays at max 760kb as measured by RSS (ps -a -o comm,rss), the maildrop is only 504kb.
Static site from template files
Posted by Carlos Daniel in Coding on September 21, 2011
Recently while working on a full-blown TurboGears based CMS for our main site it got to me the idea of creating a script to generate a simple static site from a series of Jinja2 templates, simple enough, uh? well at first I looked for a few alternatives but rapidly gave up and ended working on something of my own, as I deemed the task as a mundane and simple thing to do, it was faster creating a script than looking for one and see how it works.
Long story short, it wasn’t really that simple, there are many things to consider, such as paths, if you want to preview the output locally, relative paths will not work very well; I added a switch to my script to transform paths and a function to use on the template, suddenly I found myself adding more things like this, such as a variable for the name of the template so a menu template can check which is the current page and act accordingly.
As far as now is not that complex but a pretty handy script you could easily modify on your own, I posted the code on github:
https://github.com/clsdaniel/templatesite
As next move I would like to add a daemon mode or monitor mode so it will rebuild the output when it detects a change on the files, support for managing other files is also needed, opening the browser to preview after build would be nice, etc.
Turbogears 2.1.2 Released
Posted by Carlos Daniel in Coding on August 27, 2011
This week Turbogears 2.1.2 was released, the gears are starting to move again, the Turbogears community is getting more lively lately (as seen on the mailing list), this latest release includes a small patch I contributed to enhance Jinja2 support for Turbogears, adding the possibility of using custom filters and extensions, tough there is still a lot of work to do on that front, I haven’t had the time I would like in order to work on that, however, here at Blackchair Software we use Turbogears as one of our main technologies for business web apps, along with Django and other python related technologies (such as PySide and SQAlchemy), thus I hope to focus on contributing more on the near future.
Release announcement:
http://codersbuffet.blogspot.com/2011/08/announcement-turbogears2-212-released.html