Tim Connor's Old Blog

In my previous post, where I relate my experience using phusion/mod_rails on DreamHost PS, I was not suggesting it as a replacement for a nginx/mongrel_cluster. If you already running those happily, then you already have more control over things, and probably have everything in place to manage them. If you want a quick, easy deployment on a shared/semi-shared host, though, phusion is hard to beat. Especially since, if you are going the DH route, the single mongrel set-up of DreamHost PS requires you to juggle things around, and you’ll end up running nginx anyways.

So, in that case, for something as minor as a blog, or a minor app, it’s pretty easy to just use phusion.

Oh, and I did finally send DH a support ticket to turn my mod_rewrite back on – it’s disabled by default, since the default .httaccess people have routes to FCGI. First, though, I commented those sections out.

#RewriteRule $ index.html [QSA]

#RewriteRule ^([.]+)$ $1.html [QSA]

#RewriteCond %{REQUEST_FILENAME} !-f

#RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

DreamHost support got to my ticket unusually quickly and sure enough it was enabled… and my site was throwing 500 errors for a minute, in the apparently usual config change hiccups. Before support could get back to me the second time, it started Just Working again. So yes, mod_rewrite and mod_rails on DH PS, seem to be a-okay. On the other hand, I have not tried phusion on purely shared hosting. My venerable Typo install runs my PS up to about 150-200 megs, on average, so that’s probably a no-go for me. I suspect a small merb app wouldn’t have any problems

So, when moving my DH account over to their VPS-ish offering, I decided to stick with fcgi for a while to see how it did under less memory constrained circumstances. My blog stayed up way better, but memory usage definitely climbed consistently, and the server needed restarting every week or two. Killing off the ruby’s and the fcgi’s didn’t seem to be enough, for some reason.

So, with that as a baseline, I switched over to mod_rails. It started with an immediate jump up in memory, which had me slightly worried. And, it was a little spikier on memory usage. But, it is steady and consistent. I haven’t had to restart the module, much less my server, since I switched. It seems to be spikier (and in retrospect, they are pretty small spikes), because it actually grabs and releases memory as it needs it, unlike FCGI, which just seemed to slowly leak memory away to some hidden land where the mallocs play.

One side-note, if you are using rewrites for things like redirecting your rss to feedburner, you may need to adjust them, as the switch can mess with that. I haven’t fixed mine yet, and briefly wondered why my feedburner stats fell way off.

I’ve spent plenty of time kvetching about problems running my Typo blog on Dreamhost. I acknowledge, though, that despite some of DH’s other hiccups, the core problem is that running Rails, on Apache fcgi, on a shared host is a Bad Idea. And while I can handle the initial set-up of a slice/VPS I don’t want to deal with the maintenance. And I don’t want to pay an arm and a leg for a fully managed set-up, just to host small sites like blog.

So when my site started going down constantly, again, due to processing getting killed for being over the memory limit, most likely, I decided I would try out Dreamhost’s new “Private Server” offering, which is sort of a managed pseudo-VPS: cheaper than a true VPS, no root, they manage everything, but you do get virtualized/dedicated resources. Sounds perfect for what I need.

Unfortunately, there is a waiting list… and my site is down now! So I decide to email support and say, hey, I’ve sent some business their way over the years, my site is down now, any way to bump me to the front of the list. Well they did – but apparently side-stepping their usual processes. I show as still wait-listed in their web, panel, but all the sites go down on my account, all of a sudden, from the switch. It turns out they have moved me over to the PS, but the DNS didn’t get updated correspondingly for all my domains.

After a day or so of wrangling with their support queue, Jason got things fixed and it seems to be working. Now I just need to see how well it works for my needs. I’m going to just stick with the Apache fcgi, for now, because it works, I’m curious how the fcgi performs when not constantly being killed for exceeding the shared allowance of memory, and Dreamhost’s proxy set-up proxies you to a single mongrel by default, so I’d need to set-up up something like nginx to proxy to my mongrel clusters, and then keep all those up myself. Not that big a deal, but I am going to run with the out of the box experience a bit first, and see how little admin work I can do.

Some useful info on the DH wiki for git and on kernal.org’s guide to setting up git over http. The latter is especially helpful for debugging suggestions (like using curl to check your .netrc access permission). Using the info in these guides:

• Create a repo, or use an existing one
• Enable WebDav access on a DH directory, using their panel. Set-up the username password for the committers, and disable the password protection box for read-only anonymous access (might be easier to keep it on until you have everything set-up right)
• Create a .netrc with the auth info for this webdav
• Test your access using the curl command in the kernal.org guide (you may find it useful to have password protection enabled at this point,).
• Connect to the webdav share using the username and password.
• Copy over the .git folder contents into the root of the webdav share (this is key, because DH’s anonymous access won’t show . files)
• cd into the webdav share from your local machine and run “git-update-server-info”
• profit

You should now have access to http://yourserver/gitfoldername and http://gituser@yourserver/gitfoldername. For example: you can “git clone http://blerb.infosauce.org/git/” for read-only access to the blerb repo. Now, if I can only get gitweb working despite the limited access control imposed by DH webDAV.

Update: Changes the url for blerb to still work right after I changed it around while playing with gitweb. Also, you should use git://42squared.com/git/blerb if you actually want the real blerb repository.

Since I’ve been splitting up an app into an rails admin section and a separate camping front-end, I need to get camping working on the host to deploy. And that host, for this app, happens to be DreamHost, which means it’s gotta be fcgi. I’m not all the way done with that, but after a late night last night, I managed to get it running on fcgi locally. I’ll skip over the basics (for one thing, if you haven’t gotten this far yet, you are going to need to struggle through it to be familiar enough with what is happening to have a chance of finishing the process) until I have it all worked out smoothly, but I have to post a couple highlights, to spare some people with the same pain.

First off, no luck running off of edge. The only way I got it working consistently with mongrel, cgi (crucial for debugging and figuring out why things aren’t working), and fcgi, was to use the gem release (1.5.180), and patch that (keep reading). Once I had that running smoothly with the built-in server and then cgi, I got it working with fcgi. If you have the same problem I did, you’ll get an “`camp_do’: undefined local variable or method `exc’” error. THis is because there is another problem, which is throwing an error, but there is a bug in the exception handling code. Manually fix that (change e on to exc in the exception catching, so that it is defined), and then you can move on to the next issue, since your exceptions will now actually get through to the log, or the screen, depending on how far you’ve made it, and you can use them for more debugging.

If you have my problem, you can track down the next error to the path variable not being populated right in the fastcgi.rb

    def camp_do(req)

        root, path, dir, app = “/”

        if ENV[‘FORCE_ROOT’] and ENV[‘FORCE_ROOT’].to_i == 1

            path = req.env[‘SCRIPT_NAME’]

        else

            root = req.env[‘SCRIPT_NAME’]

            path = req.env[‘PATH_INFO’]

        end

Wether or not I set ENV[‘FORCE_ROOT’] the wrong server variables are being set for camping to know the right path. This shows up as gsub on nil errors, or “/dispatch.fcgi not found”, or some others, depending on exactly which way you are running things. This all comes back to not being able to use ScriptAlias, as in the wiki, due to only having .htaccess level access. And using RedirectRule and other mod_rewrite stuff that workins in .htaccess (I actually just stole that right from my rails .htaccess) populates things a little bit different.

My fix was to set “ENV[”FORCE_ROOT"]=1.to_s" in my dispatch.fcgi, and then change the corresponding “path =” line to match what my set-up was populating -req.env[‘REQUEST_URI’]. I tracked this down by getting things running well enough to get the default error dump to the browser, and then checking over that.

Now I need to duplicate this without using global gems (since I can’t upgrade them on the shared host), and the do more debugging on the actual host.

crontab

MAILTO=me@mydomain.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week
# |      |       |       |       |
  */5    *       *       *       *   wget -q -O /dev/null http://mydomain.com >/dev/null
  @hourly /home/myuser/clear_timocracy_sessions.sh
  @daily /home/myuser/rotate_timocracy_logs.sh

Grabs the site every 5 minutes to keep the fcgi alive on Dreamhost, and calls scripts that clear the sessions and rotate the logs

clear_timocracy_sessions.sh

#!/bin/bash

/home/mysuser/mydomain/script/runner -e production “CGI::Session::ActiveRecordStore::Session.destroy_all( [‘updated_at <?’, 1.day.ago])”

Clears out sessions older than a day.

rotate_timocracy_logs.sh

#!/bin/bash

cd /home/myuser/mydomain

/usr/sbin/logrotate -s log/logrotate.status config/logrotate.conf -f; touch public/dispatch.fcgi

Calls logrotate and then updates the timestamp on dispatch.fcgi, so Apache will create new worker processes

config/logrotate.conf

“log/*.log” {

  compress

  weekly

  delaycompress

  missingok

  notifempty

  rotate 52

  create

}

Archive weekly, keeping a years worth of logs. delaycompress – wait to gzip them, till the next week, both for ease of reading the most recent and so there is time for the fcgi threads to get killed off, since they are pointing at the old log.

After an email exchange with Dreamhost I have had it confirmed that their watcher processes monitor on a per user basis. This means that if you run each app under a separate user you are less likely to have problems with getting your fcgi processes killed.

Excerpt of my email exchange with Dreamhost
bq.

> Is it true that the processes are watched on a per user basis
> as in running each app under a different user name will
> mitigate problems?

Yep, that’s correct!

As you can no doubt see, this blog has return from its hiatus. As I’ve complained about before, running Typo on Dreamhost isn’t that dependable, so while I was busy wrapping up the sale of my condo and then moving to Tucson, I let it slide. But now that I am somewhat settled in, I’ve gotten around to the updating, tweaking, and praying required to get everything working again, and will probably be keeping a little closer eye on it.

I’ve turned on various spam protection measures (such as Akismet, disabling non-ajax commenting. moderation, etc), and am hoping that will alleviate some of the performance issues. If not I am thinking of moving onto a new system- looking for something that is less of a pain for me to maintain with this set-up, since I will be needing to get a job, and thus will have better things to do with my time than getting frustrated with Typo/Dreamhost-fcgi. I’ve been considering the newest kid on the block in the Rails blogging world SimpleLog, but maybe things will stay functional enough that I don’t have to go down that road and spend more time converting everything over.

Either way, I’ll be posting again here some, as I work on finally getting the Lost River Outfitter’s site updated with the new functionality that I promised them as a Christmas present ages ago, and then finally get around to playing with some of my ideas for InfoSauce 2.0. If I get a non-tech job (like my plan to work in a bike shop), I’ll probably be willing to spend more of my spare time developing, but either way I’ll try and post semi-regularly as I come across cool stuff.

As well, I’ll probably be working on getting the design updated with Stf’s work (after I feedback to her on the first round logo stuff), so I can get rid of the stupid placeholder header image, fixing the homepage nav in Safari, putting my resume up, and other house-keeping as I have time.

So I thought I’d hold onto Dreamhost as my primary host a while longer, but its driving me mad. It just won’t perform wortha snot, especially anything that depends on fcgi. Hence Trac and Typo (Rails is Apache fcgi on Dreamhost) are not exactly running up to snuff. But it’s not like anything has ever been going F1 speeds for me with them.

You want to know how I knew right away there was a comment posted on one of the entries? I have been giving the new Rails monitoring service, Heartbeat, a try, and I got a notification from them that my domain had changed from “200 OK” to timing out. Great, the load of serving up a couple page-views and handling a single comment pushed my Typo install to it’s limits.

I’m going to keep Dreamhost for storage, backup, and mirroring svn, since you get a lot for your dollar, but I can’t stand it as a primary domain any longer, even just for my blog and minor personal projects.

Dreamhost upgraded the RAM on my server yesterday. This meant it was down unexpectdly and I felt like that was teh last straw. BUT, I checked their special status page, and saw what was happening, and they did have it right back up around the time frame they said they would.

My sites perform MUCH better now. I’m still stuck with the vagaries of being on a shared host, with the (mt) gs set-up might be the solution for, but so far I haven’t had any problems. Maybe half my problem was just being stuck on an underequiped server. And after trying out the gs, I might be asking for a refund (they said they’d give me one, if I chose). I’m just too addicted to the featurefulness of DH. mt didn’t have SRV records (needed for the chat for GMail for Your Domain), email plus addressing (tim+randomstring@timocracy.com), anyway to run Trac, barring good (bad) old cgi*, or mod\dav\svn (svn over Apache).

The solution for most of these would be too keep my DH account for those and use the Grid for things like Rails sites that need better performance. Well, why not just keep my DH only then, since my sites are performing fine now. I can always go back to mt later, after they add a Python container and SRV records (two things I know they are planning to implement). Thanks mt, for making it easy to cancel and get a refund, I’ll remember that when I need a little more umph, or if DH starts sucking again.

*While Apache fcgi may suck, it’s better than that.