Tim Connor's Old Blog

I rarely update this blog, possibly because dealing with Typo is more than I can bothered to do anymore. So I am caching it to HTML, and moving on to newer lighter-weight approaches that require no maintenance from me. This blog will be moving to http://oldblog.timocracy.com and a redirect put in place for all the old content. http://www.timocracy.com will continue to be my home on the web; I may just set-up a redirect (or maybe proxy everything) in Apache to my tumblog.

So, once I added a couple Rails metal to our high volume site, my boss complained that our performance took a serious hit. In tracking it down (with strace) we realized that every request was re-requiring each metal, which involves a full load path search. This was not happening on our local script/server started mongrels, but was happening on production for both thin and mongrel_cluster, even after we had gone through every setting and made them match exactly.

We traced it down to the difference of mongrel_cluster versus the script/server. Script/server is going to use the built in, updated for Rails 2.3, mongrel handler. The old stuff called by thin or mongrels started by mongrel_cluster, is going to use the old (“deprecated”) dispatch path, which in a cgi-ish fashion rebuilds the dispatcher and all rack middleware and metal each time, which means re-requiring them all.

So now I am not sure what to do. Do we have to start all of our mongrels with rackup? Do we take the rack mongrel handlers from rails and patch them into wherever? Do I file a bug for Rails that there at least needs to be some documentation, other than just deprecating the method that a stock mongrel will call?

Update I made a ticket for it Also having mongrel start up using the handler in the rails vendorized rack as a config will fix the problem for mongrel. I’m not sure about thin, as that handler seems to be rackup specific?

Update2 Ezra said he’d go with rackup and gave me a monit script for it

Update3 We made a barebone rails config.ru and made thin use that, forcing it down the path of using the built-in rails handler. That dropped our load by 30% roughly, so yeah, this behavior had a significant impact on us.

In my previous post, where I relate my experience using phusion/mod_rails on DreamHost PS, I was not suggesting it as a replacement for a nginx/mongrel_cluster. If you already running those happily, then you already have more control over things, and probably have everything in place to manage them. If you want a quick, easy deployment on a shared/semi-shared host, though, phusion is hard to beat. Especially since, if you are going the DH route, the single mongrel set-up of DreamHost PS requires you to juggle things around, and you’ll end up running nginx anyways.

So, in that case, for something as minor as a blog, or a minor app, it’s pretty easy to just use phusion.

Oh, and I did finally send DH a support ticket to turn my mod_rewrite back on – it’s disabled by default, since the default .httaccess people have routes to FCGI. First, though, I commented those sections out.

#RewriteRule $ index.html [QSA]

#RewriteRule ^([.]+)$ $1.html [QSA]

#RewriteCond %{REQUEST_FILENAME} !-f

#RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

DreamHost support got to my ticket unusually quickly and sure enough it was enabled… and my site was throwing 500 errors for a minute, in the apparently usual config change hiccups. Before support could get back to me the second time, it started Just Working again. So yes, mod_rewrite and mod_rails on DH PS, seem to be a-okay. On the other hand, I have not tried phusion on purely shared hosting. My venerable Typo install runs my PS up to about 150-200 megs, on average, so that’s probably a no-go for me. I suspect a small merb app wouldn’t have any problems

So, when moving my DH account over to their VPS-ish offering, I decided to stick with fcgi for a while to see how it did under less memory constrained circumstances. My blog stayed up way better, but memory usage definitely climbed consistently, and the server needed restarting every week or two. Killing off the ruby’s and the fcgi’s didn’t seem to be enough, for some reason.

So, with that as a baseline, I switched over to mod_rails. It started with an immediate jump up in memory, which had me slightly worried. And, it was a little spikier on memory usage. But, it is steady and consistent. I haven’t had to restart the module, much less my server, since I switched. It seems to be spikier (and in retrospect, they are pretty small spikes), because it actually grabs and releases memory as it needs it, unlike FCGI, which just seemed to slowly leak memory away to some hidden land where the mallocs play.

One side-note, if you are using rewrites for things like redirecting your rss to feedburner, you may need to adjust them, as the switch can mess with that. I haven’t fixed mine yet, and briefly wondered why my feedburner stats fell way off.

I’ve spent plenty of time kvetching about problems running my Typo blog on Dreamhost. I acknowledge, though, that despite some of DH’s other hiccups, the core problem is that running Rails, on Apache fcgi, on a shared host is a Bad Idea. And while I can handle the initial set-up of a slice/VPS I don’t want to deal with the maintenance. And I don’t want to pay an arm and a leg for a fully managed set-up, just to host small sites like blog.

So when my site started going down constantly, again, due to processing getting killed for being over the memory limit, most likely, I decided I would try out Dreamhost’s new “Private Server” offering, which is sort of a managed pseudo-VPS: cheaper than a true VPS, no root, they manage everything, but you do get virtualized/dedicated resources. Sounds perfect for what I need.

Unfortunately, there is a waiting list… and my site is down now! So I decide to email support and say, hey, I’ve sent some business their way over the years, my site is down now, any way to bump me to the front of the list. Well they did – but apparently side-stepping their usual processes. I show as still wait-listed in their web, panel, but all the sites go down on my account, all of a sudden, from the switch. It turns out they have moved me over to the PS, but the DNS didn’t get updated correspondingly for all my domains.

After a day or so of wrangling with their support queue, Jason got things fixed and it seems to be working. Now I just need to see how well it works for my needs. I’m going to just stick with the Apache fcgi, for now, because it works, I’m curious how the fcgi performs when not constantly being killed for exceeding the shared allowance of memory, and Dreamhost’s proxy set-up proxies you to a single mongrel by default, so I’d need to set-up up something like nginx to proxy to my mongrel clusters, and then keep all those up myself. Not that big a deal, but I am going to run with the out of the box experience a bit first, and see how little admin work I can do.

Some useful info on the DH wiki for git and on kernal.org’s guide to setting up git over http. The latter is especially helpful for debugging suggestions (like using curl to check your .netrc access permission). Using the info in these guides:

• Create a repo, or use an existing one
• Enable WebDav access on a DH directory, using their panel. Set-up the username password for the committers, and disable the password protection box for read-only anonymous access (might be easier to keep it on until you have everything set-up right)
• Create a .netrc with the auth info for this webdav
• Test your access using the curl command in the kernal.org guide (you may find it useful to have password protection enabled at this point,).
• Connect to the webdav share using the username and password.
• Copy over the .git folder contents into the root of the webdav share (this is key, because DH’s anonymous access won’t show . files)
• cd into the webdav share from your local machine and run “git-update-server-info”
• profit

You should now have access to http://yourserver/gitfoldername and http://gituser@yourserver/gitfoldername. For example: you can “git clone http://blerb.infosauce.org/git/” for read-only access to the blerb repo. Now, if I can only get gitweb working despite the limited access control imposed by DH webDAV.

Update: Changes the url for blerb to still work right after I changed it around while playing with gitweb. Also, you should use git://42squared.com/git/blerb if you actually want the real blerb repository.

So I’ve got some spam that is being submitted through AJAX (I have non-Ajax comments turned off) and slipping through akimset. One could think maybe it’s being hand-crafted, because it sometimes makes sense on a tech blog, but it isn’t pertinent to the actual specific article. I guess they know use a database of tech comments, maybe scraped off from existing comments, and apply them willy nilly on blogs that cover these topics, while setting the url field for the name to their spammy site? Why don’t I think it’s hand-crafted?

If LogoMaid made your logo, you are supporting trademark theft and copyright violation has a comment “why it’s always writing NULL in empty fields?” and I forgot how much I really hate shell scripting has “Predicted values in template editing can be done through any PHP script?”. So they sorta make sense, just not as comments on these articles, or even on any of mine I can think of. Database of techy comments for use in spam?

Neither Analytics or Feedburner have directions for attaching their scripts “properly:” in the head element and without in-lined javascript. Yet both of them work fine, when you do.

For feedburner, just follow the directions for the simplestats, but put the script tag in the head, not in the body, as directed:

For Analytics, attach the script, and use some method of calling a cross-browser DOMContentLoaded. I, use the one from lowpro since I’m mainly developing in Rails, so using Prototype is a no-brainer.

While properly setting up my Rails app server maintenance tasks, I noticed that my Typo install was running even more bloated than I remembered – no wonder the Dreamhost watcher processes kill it off. After digging around and trying a couple things it seemed that deleting your unused text-filters, sidebars, and plugins does nicely, for trimming down Typo.

Well, I got my Typo install down from running in the 50s to 30s (megs)upon the initial start (it still grows as it leaks, of course). See the glory of what I hacked out of the vendor folder.

!      vendor/bluecloth

!      vendor/plugins/magnolia_sidebar

!      vendor/plugins/haml

!      vendor/plugins/aimpresence_sidebar

!      vendor/plugins/fortythree_sidebar

!      vendor/plugins/technorati_sidebar

!      vendor/plugins/typo_textfilter_tmcode

!      vendor/plugins/tada_sidebar

!      vendor/plugins/typo_textfilter_lightbox

!      vendor/plugins/typo_textfilter_markdown

!      vendor/plugins/tiny_mce

!      vendor/plugins/xbox_sidebar

!      vendor/plugins/typo_textfilter_amazon

!      vendor/plugins/typo_textfilter_sparkline

!      vendor/plugins/amazon_sidebar

!      vendor/plugins/recent_comments_sidebar

!      vendor/plugins/typo_textfilter_textile_and_markdown

!      vendor/plugins/audioscrobbler_sidebar

!      vendor/plugins/backpack_sidebar

!      vendor/plugins/xml_sidebar

!      vendor/plugins/typo_textfilter_flickr

!      vendor/plugins/typo_textfilter_code

!      vendor/plugins/flickr_sidebar

!      vendor/plugins/typo_textfilter_smartypants

!      vendor/plugins/upcoming_sidebar

!      vendor/plugins/sitealizer

!      vendor/plugins/fortythreeplaces_sidebar

!      vendor/rubypants

!      vendor/flickr

!      vendor/jabber4r

!      vendor/sparklines

Here are the corresponding environment.rb changes. I also deleted all but the Textile and none filters in the admin interface. The first thing I tried chopping out, though, was the newly added sitealizer. Since this is run via an around filter, you have to edit your application.rb thusly, if you take it that far. I, also, got carried away and took out BlueCloth, since Textile is handled by RedCloth. In doing so, I found at least one place in the admin interface were it is hard-coded, and will cause your new content form to choke: app/views/admin/content/_form.rhtml

UPDATE – five minutes later: I forgot, I also turned off live Preview to stop hammering my poor app, and started editing directly in Textmate, from the textarea

UPDATE – 4/19/2007 – Found another hard-coded instance of BlueCloth that I had to comment out:
app/models/text_filter.rb/comment_help line 96

f.help_text.blank? ? ’’ : help_text  # “#{BlueCloth.new(f.help_text).to_html}\n”

crontab

MAILTO=me@mydomain.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week
# |      |       |       |       |
  */5    *       *       *       *   wget -q -O /dev/null http://mydomain.com >/dev/null
  @hourly /home/myuser/clear_timocracy_sessions.sh
  @daily /home/myuser/rotate_timocracy_logs.sh

Grabs the site every 5 minutes to keep the fcgi alive on Dreamhost, and calls scripts that clear the sessions and rotate the logs

clear_timocracy_sessions.sh

#!/bin/bash

/home/mysuser/mydomain/script/runner -e production “CGI::Session::ActiveRecordStore::Session.destroy_all( [‘updated_at <?’, 1.day.ago])”

Clears out sessions older than a day.

rotate_timocracy_logs.sh

#!/bin/bash

cd /home/myuser/mydomain

/usr/sbin/logrotate -s log/logrotate.status config/logrotate.conf -f; touch public/dispatch.fcgi

Calls logrotate and then updates the timestamp on dispatch.fcgi, so Apache will create new worker processes

config/logrotate.conf

“log/*.log” {

  compress

  weekly

  delaycompress

  missingok

  notifempty

  rotate 52

  create

}

Archive weekly, keeping a years worth of logs. delaycompress – wait to gzip them, till the next week, both for ease of reading the most recent and so there is time for the fcgi threads to get killed off, since they are pointing at the old log.