IX, down again

This site used to live on IX web hosting. I'm not sure what happened at IX, but 2009 hasn't been a good year for them. I ran six web sites there and they seemed to be  more down than up, or at least that's how it felt. My monitoring service suggested that towards the end they were as bad as 80% available on a week-by-week basis. That's bad enough, but even worse were their "customer services"

Everyone at IX support is desperately polite but completely obtuse. "Your asp.net compiler is broken, and here's the crash dump from it" I would say. They would apologise profusely for the difficulty and then claim it was fixed, occasionally alluding to some undefined technical problem which they had sorted out.  Other times they'd claim that the problem was with the sites themselves, claiming for example that BlogEngine breaks asp.net! Actually the problems were intermittent but repeatable, and as far as I can tell they didn't do anything at all about them.

When questioned more closely IX were evasive, perhaps hiding behind their odd usage of English (it's an American company, but the people working there appear to be from elsewhere). If this was real work I'd have sacked them the second time they did that to me; as customer I didn't have that luxury.

IMAPsync for Windows

 

IMAPsync's install instructions are here.

In summary, here's what you need to do. I optimized the order. All this stuff is free and doesn't nag you. IMAPsync is a program which requires Perl to run, so you need to install Perl then IMAPsync. Finally you need to test your install, and then write a script to move your users' email.

1 Install Perl

  • Download ActivePerl and install it with the default options. It's basically a command line tool, all you get in your program menu is the "Perl Package Manager", which you need to run next.
  • Run Perl Package Manager. This lists all the add-ons installed with Perl. There's a lot of stuff already, but you need to add "Mail-IMAPClient". Tell the PAckage Manager to display "all available packages", as opposed to only installed packages, the default. Type in "Mail-IMAPClient" and you'll see the package in the list. Select it, mark it for install, and install it. it will drag in some other things it needs.

Ok, now you have a basic Perl install with the IMAP interface.

2 Install IMAPsync

  • Download the latest distribution of IMAPsync from here. You'll immediately notice that it's a tgz file, something you can't easily open on Windows. I couldn't find an online service to unpack it, so...
  • Download PeaZip, which opens tgz files. Install this and let it handle any archive extensions windows doesn't handle by default - tgz and all those old unix style files.
  • Open the IMAPsync tgz file, and extract the contents. Which is another compressed file. Use PeaZip again to extract that, and you'll have a directory called imapsync with the goods inside it.
  • Put the imapsync directory somewhere handy. As Perl installs itself in c:\Perl, so I put imapsynch in there: c:\Perl\imapsync.

3 Test IMAPsync

Open a Command Prompt and change directory to c:\Perl. Type "perl -v", which returns version information from Active Perl. Now you know your Perl's good.

Assuming you put your imapsync where I put it above, then typing: "perl imapsync\imapsync" will return the help information for imapsync.

4 Script some email copies

IMAPsync is fairly clear on what the options do. Typically you want to copy mails one way, from the old system to the new. You usually need to do this multiple times. The first time it'll take a long time because all the mail has to be transfered; the second time will be quicker as only new mail will move. Hence you'er going to want to script this with windows batch (BAT) files.

Try something like:

perl imapsync\imapsync --host1 mail.loser.com --user1 phil@loser.com --password1 lkj!jh455 --host2 imap.winner.com --user2 phil@winner.com --password2 jhy9$lk --dry

The "--dry" means it will connect, proving that you have user names, passwords and hosts correct, but it won't actually copy mail. Once you have that working, stick it in a batch file and remove the "--dry", and run it until both hosts are in synch.

I must have had several hundred of the most fulsome (in the true sense of the word: they didn't mean any of it) apologies over the last few weeks. I'd rather have had one competent technician, or even an idiot who would reboot the machine when asked.

I suspect that the support organization at IX is completely separate from the operational side. There was no obvious sign that anyone I talked to there had ever touched a machine, or knew how to. They didn't know, for example, the difference between serving plain old html and serving aspx. I saw no evindence of any technical knowledge from the support staff. Even basic things like how to examine server logs were simply not understood by the support staff. They appear to work from scripts which they interpret with little technical knowledge. That approach may work for general non-technical stuff, but when they have a machine which is clearly unstable, these guys are the wrong people to deal with the problem.

Obviously as someone with a fair amount of customer-facing technical support experience I tried every route I could get to someone useful. If there'd just been someone in IX with root access I could have told them how to fix things. It would have been cheaper for IX to fix the problems I suffered rather than deal with my complaints, but it's their business. I have other sites with other mass-market hosting companies who do seem able to run asp.net servers: my expectations are not unreasonable. Ya buy stuff, it ought to work most of the time and the support people shouldn't tell you lies.

I generally charge for this type of consultancy, but my for-free advice to IX is: get yourself some decent senior management or sell your customer base before it implodes on you.

So there you go, welcome to "Mosso", "cloud computing" and IIS7 (about time!).

Things to help you escape IX Webhosting

You're no longer going to need POP before SMTP (doh!).

SQL server database migration

IX are pretty useless at database backups, and although they'll take a backup to help you set your server up with them in the first place, they won't give you back your data when you're ready to leave.

You can't get to your data with SQL Server Management Studio Express (who thinks up those names?), which is a bit of a worry. But you can get to it with the Database Publishing wizard, which currently resides here. That's one of those where you download it and try to install it, and then it tells you that you need a bunch of other things. That will happen a couple of times, but stick with it as it does work in the end. I use the wizard to upload schema changes and stored procedures anyway, so it's pretty straightforward to use it to download schema and data as a full backup. Watch the settings closely though, as the word "drop" does occur in there somewhere. The wizard produces a SQL file, which you'll need to feed into Management Studio Express to run it against your new database.

An alternative approach would be to code a stored procedure to backup the database to the website's file space, then suck that out via ftp.

One thing which is kind of handy is that hosting services' databases, like their web servers, are 'net accessible. Say what? Well, you can access an IX database from your desktop PC to back it up using the database publishing wizard as above. In the same way, you could access that same database from another hoster's asp.net server. So you can migrate your web server first, then shift the database later. This is actually precisely what you want to do, as you can't easily control when the DNS cuts over... 

Just:

  1. Test and run identical web front ends on both the old and the new server, both referencing the same old database.
  2. Cut the domain over, and wait a few days before taking the old server offline
  3. Use the database publishing wizard to copy your database from the old to the new server. Check how long it takes: that's the size of the outage you need.
  4. Take the new front end offline (so no one's accessing the database) and repeat (3).
  5. Change the database connexion string to point to the new database instance and put the front end back on line.
  6. Test and regress as required.

 

I had a lot of databases to shift so I found a few interesting things whilst doing it...

error 4064

First, some of your logins will have strange default databases set for them and you can't connect to them with the Publishing Wizard as a consequence. It barfs with error "4064" when you try. I wasted a fair few minutes on those. The solution is to connect to the database "master" (which is always there and belongs to IX), then change the default database for the user you are logged in as to the database you're trying to publish:

(thanks to for this one).

encryption problems

In order to protect a database from being stolen, it has to be encrypted. Unfortunately "being stolen" is quite similar to "being moved to another host", if you see what I mean. Ideally you'd just backup and restore the database, but IX don't let you do that. You could extract (back up to file) the various keys you need, but again IX won't let you do it. In the end I wrote some code to decrypt the stuff I needed, then re-encrypted it on the new machine. There may be a better way; if there isn't, then this is a weakness of the IX-style shared hosting.

email migration

This is an issue for moving from anywhere to anywhere. As someone commented in surprise on another post here: "you're an expert". It's slightly embarrassing, but actually I kind of should be. Anyway, a tool which gets a lot of professional use on massively scaled systems is imapsync. Telcos and the like use it to move millions of users from one email provider to another, without dropping anything. The "linux" bit in the URL is a clue that it's going to be a bit of a pig to get going, but it is installable on Windows and works fine there (see sidebar). If you follow the instructions, install Perl and then the other bits, it then does a nifty job of synchronizing imap accounts between your old and new hosts. Just set up a batch file and keep running it until your domain and users are cut over, then stop.

The problem of course is that you will need a list of user passwords, but you're going to need that in order to set up the new accounts anyway.

Mosso's hosting

Some notes on what to expect at Mosso are here.