Recently, I've been toying with the idea of setting up a couple extra services on my guild's Domino server - voice chat, non-Sametime chat, what have you - and I figured I should give a shot to LDAP authentication with the Domino directory for these. However, this is something I've never done, and the documentation is a little rough - most LDAP info on the web refers to non-Domino servers, while most Domino-specific information was written in about 1996.

I'll leave out the depressing details of the various things I tried in my quest to get LDAP working as an authentication mechanism for my Linux server (as a relatively simple test case) and point you instead to this dead-but-still-archived page: http://web.archive.org/web/20040614140723/http://www.dominux.co.uk/ldap.html. The key information on that page is the list of fields that you have to add to your user documents to use them for this purpose. During my harried testing, all /var/log/auth.log was telling me was "Invalid credentials", but what it really meant was that the user account it found didn't have the right attributes. Thanks, Linux!

Though most of my Domino programming has been done in LotusScript (since it's one of the two "real" Domino languages), I had worked with Java here and there before diving into XPages, at least enough to know about recycle(). recycle() is a strange beast, a visitor from a non-memory-managed language popping up inside a famously memory-managed one. I get, conceptually, why it exists - since Lotus doesn't control the Java object lifecycle, Domino can never know when an object is garbage collected. And I'm sure there was some efficiency- or pragmatism-related reason why the back-end C++ objects were necessary when the Java API was created, but the result is that there's this weird anachronistic headache to deal with.

In the case of Java agents, it's not so bad - most of the time, the agent will be very procedural and so it's very easy to toss in some recycle()s in your loops and at the end of the code. However, it's much stickier with XPages, especially if you have a crazy back-end object system like I do that's meant to abstract away all of the implementation details of the Domino database. Once you reach that point, it's very hard to have code "know" when an object is no longer going to be needed. It becomes this balancing act between strict recycling on the one hand (resulting in many more round trips to the database than needed) and fast but leaky code on the other.

However, though each individual bit of code doesn't necessarily know if it's at the end of the lifecycle, there IS a well-defined set of lifecycle phases and a mechanism for hooking into that. Taking a note from how to implement a flashScope in XPages, I created a new view-scoped backing bean that inherits from a thread-safe Set containing lotus.domino.Base objects and adds a convenience method to call .recycle() on all of its contents. Then, I added a PhaseListener object to wait for after the "Render Response" phase and call that. Then, everywhere in my code that I create a Domino object, I add it to the Set before continuing along in the code. Since I'll definitely be done with all of those objects by the time the page has finished rendering, this should theoretically mean that all of my objects are recycled after each page load.

Recently, my living room Mac mini's external drive, where it kept its iTunes library, met a horrible, clicking death. Naturally, I had devised a proper backup plan long before this happened - unfortunately, however, I had not yet implemented this plan. Crap.

So I decided to straighten out my backup system all around, ideally covering all of the machines in the apartment as well as my hosted Domino server. My ideal (at least for now) backup plan would fit within a couple attributes:

• Cheap. I've been trying to spend less money overall lately, and picking up a new bill for hardware or services would counteract that somewhat. I wanted to come up with something that would work with the hardware I had on hand, plus one purchased replacement HD.
• Automatic. I don't want to have to remember any manual backup process, since I most likely wouldn't.
• Off-site backup isn't important. Sure, it would be nice to keep my data in the event of a physical catastrophe, but we're talking about TV shows and movies here, not anything vital.
• Quick recovery or automatic failover aren't important. They'd be NICE, certainly, but I'm just looking for a basic "the data exists in at least one other place" setup. If a computer meltdown means that recovery will take a while or I'll have to rebuild the OS, that's fine.
• Versioning isn't important. The occasions where I would want to restore intentionally-deleted files or modified documents are so few and far between that it's not worth going out of my way to achieve that.

My main laptop was far and away the easiest to set up. A while ago, my boss gave me an external USB drive which I've been using for Time Machine backups. Time Machine does pretty much everything I would want to, and even gets bonus points for ease of recovery and versioning. When I'm out of the office, it even kind of counts as off-site.

My Domino data was the next easiest, primarily since I work with it all the time. I set up a Parallels virtual machine to run a new Domino server, set up scheduled replication, and pointed my "create a replica of everything" agent at my production server. Voilà: up-to-the-hour backups without having to give it a second thought.

The hard drive I purchased to replace the failed one was a nice 2 TB one, giving me enough room to store my media library plus some Time Machine backups for the mini itself, the iMac in the bedroom, and the two other laptops floating around. It won't be enough space permanently, but it'll last me at least until I'm comfortable enough to buy another one. So that covers the other Macs themselves.

In addition to the media drive, the Mac mini also has a 750GB drive salvaged from my poor, video-card-exploded iMac. I cleaned off enough old crap from there that it will be able to serve as a mirror for the media files on its larger brother - again, at least for now. To implement that, I wrote a quick, two-line shell script:

rsync -aE --delete /Volumes/Tartaros/Movies /Volumes/Diaspar
rsync -aE --delete /Volumes/Tartaros/iTunes /Volumes/Diaspar

That basically mirrors the Movies and iTunes folders on Tartaros (the media drive) to equivalent folders on Diaspar (the iMac's old drive). The "-a" switch toggles "archive" mode, which enables a lot of useful behaviors for this case, "-E" enables support for HFS+ metadata like ACLs, forks, and extended attributes, while "--delete" removes any files in the target directory that no longer exist in the source. I added this to my crontab to run at 3 AM each day.

All in all, I think this setup should cover my basic needs pretty well. My next step, when I want to spend the money, will be to sign up for an online backup service like CrashPlan. There are some cheap options and they would hopefully be more reliable than my current scheme, which is still dependent on the fragile health of a handful of external USB drives.

Mostly on a lark but partly for practical reasons, I wanted to set up PHP on my Linux-based Domino server. While an easy option would be to install Apache and have it use a non-80 port, I wanted to see if Domino's ancient CGI support still works in the modern world of today.

In short: yes. In fact, it's pretty easy; the bulk of the work I put into it was actually just because I refused to believe that the syntax on that IBM page and in Administrator's documentation wasn't a typo. On my Ubuntu-based server, the setup is quick:

• Run "apt-get install php5-cgi"
• Set "cgi.force_redirect = 0" in your php.ini file. In my case, this didn't exist, so I created it as /etc/php5/cgi/conf.d/php.ini with just that line. If you don't do this, you'll get errors about the CGI app not returning any output
• Add a rule to your web site with the following setup (using the path to your PHP CGI binary as needed):
  • Type of rule: Directory
  • Incoming URL pattern: /*.php
  • Target server directory: /usr/bin/php5-cgi\*.php
  • Access level: Execute
• Place your scripts in the domino/html directory in your server's data directory

That backslash is actually there and there actually isn't any space in front of it. It's really weird, I know.

If everything went well, you should be able to refresh HTTP and voilà: you can execute PHP scripts. I haven't investigated it super-thoroughly for edge-case gotchas, but it runs phpinfo() just fine, and you don't have to worry about making your PHP scripts executable or anything.

One important thing to bear in mind is that this is just CGI. Your PHP code isn't going to suddenly have access to the Domino API in any new way - this isn't a new way to write Notes apps on the web. If you want some sort of API access, on Windows you could use OLE and on other platforms you could presumably compile a version of PHP with Java support, but that's about it. That said, you DO get the benefit of Domino handing off any authenticated user name (even using some crazy DSAPI method), so it's not COMPLETELY isolated from the rest of the server.

Periodically (read: every day), I wonder about switching from Domino to a SQL server for data storage on my guild web site. The primary reason for this is speed: I'm doing primarily relational things, so I've had to wrangle Domino quite a bit to do this with any amount of speed and code cleanliness. Additionally, while most of my documents are entirely distinct from each other, I had to make concessions here and there, such as storing the latest Post date in Topic documents so I can sort them that way, and, each time I have to do that, there's another little bit of code maintenance and clustering-unsafety.

However, my ideas always come to a screeching halt when I remember Reader fields. They're simply too good, and the replacements I've found on the open source SQL databases have been, to put it kindly, lacking in comparison. They generally involve having some access level field or, best case, a multi-value field of names that are allowed to see the document, and then making sure that all of your queries or views honor that. Every method has some severe downside, ranging from inflexibility (access level) to nightmarish piles of code everywhere (multi-value name/group/role fields). Everywhere I accessed the database, I'd have to worry about security and document access, bloating up the code and just asking for data-leak bugs.

Domino, for all of its faults, makes this something you just don't have to worry about. If you have a Reader field, you can toss names, groups, and roles in there with impunity, and the server will handle the rest like you'd want. You don't have to do your own directory lookups, security checks, or nested queries. If the current user isn't on the list, the document may as well not exist. Even if the user had the UNID of the document and designer access to the database, it'd be beyond their reach. This is enormously comforting. And even though it's just a guild web site and not a giant corporate database, I'd still rather deal with a bit of tricky code for performance than the headaches and drama involved with people seeing what they're not allowed to see.

So, until I either get entirely fed up with Designer or I find an equivalent to Reader fields in a free SQL server, I'll be sticking with Domino.

Around 8:45 PM last night, my main XPages app stopped responding. The browser would sit there waiting for the server for about 30 seconds or a minute until the server finally gave up and handed out a Command Not Handled Exception. When I first started looking into it, I saw a rogue process taking up the whole CPU, but after killing it and bouncing the server for good measure, the problem remained.

I'll leave out the hour's worth of hair-pulling and cut right to the chase: I had added a doctype line to my faces-config file but hadn't gotten around to removing it. This normally isn't too much of a problem, but java.sun.com became unavailable yesterday (and still is at the moment). Thus, the server was opening the faces-config.xml file as XML and, as XML parsers are supposed to it, it was attempting to fetch the DTD to validate it. However, after waiting 30 seconds or so, it would give up the ghost, spit a misleading error to the console along the lines of "Can't parse configuration file:xspnsf://server:0/database.nsf/WEB-INF/faces-config.xml", and declare that it couldn't handle the command. As soon as I removed the DTD, everything started working perfectly again.

I'm reasonably certain this is Larry Ellison's fault.

The main non-work project I've been working on for the last... forever, it seems, has been a forum app for my guild. It started out as a raid-composition utility, then grew to have loot handing, and then added a forum. Most of it is pretty WoW-specific, but the upshot of the whole thing is that it's a moderately-complex XPages application, which has done wonders for my knowledge of the environment.

For programmatic convenient and, in some cases, performance, I've written a ton of wrapper classes, such that I almost never use the standard <xp:dominoDocument/> and <xp:dominoView/> data elements directly, instead using Java beans and collections that implement List. This allows me to do things like #{post.topic.forum.title} if I want to without having to worry about the XPage knowing about how to fetch each parent document in a view.

However, I've run into an annoying problem. Namely, this problem, wherein the Java classloader gets confused about different versions of some classes and stubbornly declares that a class is incompatible with itself, even when I didn't change anything with the class. From what I gather, this is something that you're sort of bound to run into with XPages once you start mucking about with scoped beans enough. It's gotten to the point in my app where I can trigger this behavior 100% of the time by simply re-saving any XPage document. Fortunately, I can clear it up by re-saving a Java class file, but that means that every change I make to an XPage or a Custom Control has to be followed by a re-save of a Java class, which is a step I'd rather not have to take. Worse, it seems to also happen when Domino document data is modified via a mechanism other than the XPages themselves (say, via replication), which is kind of a serious problem.

For the most part, I've learned to work around this. When developing, I re-save a Java class file after each XPage, and I (horrifyingly) wrote a cron script to check the page I know exhibits this error when it happens every 15 minutes and restarts Domino as necessary (which is why I wrote the special login DSAPI filter). These are terrible things to have to do, though, and so I'm really trying to figure out how to actually fix the problem.

Clearly, something I'm doing in my code has caused this, since it doesn't happen in my other apps, but it's not clear what it is and if it's really something I'm doing wrong. I've tried switching all of my managed beans to request scope, implementing Serializable all over the place, changing the way XPages are cached, fiddling with the compiler target version in the Eclipse project, eliminating all Domino classes from object instance variables, installing server fix packs as they come out, and, heck, switching from Windows to Linux, just in case.

At this point, my workarounds pretty much work, but it still drives me nuts, especially the server-bounce script. Since I suspect the culprit is related to all the managed beans, I'm going to try cutting down the data-type collections (Posts, Topics, etc.) and replacing them with <xp:dataContext/>s on the pages that use them, but I don't have terribly high hopes for that. Maybe whatever is causing it on the server side will be fixed in 8.5.3, but I'd still love to fix it before then. I may have to resign myself to it, though, and hope the problem does indeed go away once I'm no longer actively working on it.

For my main project, which contains a forum, one of the problems I ran into was Domino's session handling. Namely, it's designed such that HTTP user sessions last for the duration of the browser session and time out after 30 minutes. That's fine for, say, a corporate app, where you don't want a client logged in indefinitely. However, having to re-log-in to a forum every time you visit it would be a hassle.

The session timeout is easy to fix - you can just up the timeout period in the web site config. The cookie took a little more trickery, but wasn't too bad either. I set up some code in the beforePageLoad event to look for a DomAuthSessId cookie and, if present, create a persistent cookie. The server can't tell the difference between the session and persistent cookies, so this works.

However, it's sort of ugly. While I absolutely don't want to have people re-log-in every time they restart their browser, I also don't really want tons of session-scoped managed beans floating around for months, and this kind of trick defeats any use of the Internet Users list in Domino Administrator (a minor quibble, but a quibble nonetheless). Ideally, I'd be able to automatically log people in on subsequent visits, but not have to actually maintain server sessions for them. From all I've read, this calls for a DSAPI filter. Since that involves C/C++, there's bound to be a lot of setup and tribulation.

First off, I had to grab the Notes/Domino C API toolkit. Fortunately, Google led me directly to the IBM download page and, after re-confirming that I don't want them to send me email, I was given a download link for a quaintly compress'ed file and some installation instructions. I dropped the libraries into /opt/ibm/lotus/notesapi and set up a symlink at /opt/lotus/notesapi just in case. The samples in the toolkit do indeed include a DSAPI filter, and I found another one doing something very much like what I want here:

http://www-10.lotus.com/ldd/46dom.nsf/55c38d716d632d9b8525689b005ba1c0/dd8cc7c9ad12887a85256bca003476bf?OpenDocument

To keep things simple, I started with that code and commented/deleted everything that actually does something - I just wanted the simplest possible filter to see if it works at all. All it does is announce that it's alive and then proceed to not handle any requests:

dsapilogin.c

Now came the issues. First off, the DSAPI example in the toolkit only came with a handful of Makefiles, and none for Linux. The Solaris one was close, so I borrowed and modified that one with some tips from http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/fbd4bb196ce4146685256df10038a0b9?OpenDocument . The most vital thing was that, since I'm running in a 64-bit environment but using a 32-bit Domino server, I had to go out of my way to compile a 32-bit library - this is done by adding "-m32" to the CCOPTS and (maybe required) LIBS lines. The Makefile I ended up using is:

Makefile

Along the lines of the "-m32" flag, I had to install some additional development libraries to support 32-bit compiling. In my travails, I ended up installing all sorts of apt packages before it worked, so it's possible that only the last (or another, non-C++ library) is necessary:

g++
lib32stdc++6
libc6-dev-i386
g++-multilib

Unfortunately, even when you get the library building properly, it's tough to get it to load into Domino, and it's not very helpful as to why - all it says is "Failed to load DSAPI filter: " and then the file name... and it gives the same error if you point it to a non-existent file, so there's no way of knowing WHY it failed to load it. It's all trial and error (unless you actually know what you're doing, presumably).

Once all that was in line, though, I could build the library with make, copy it to /opt/ibm/lotus/notes/latest/linux, add "libdsapilogin.so" to the Configuration tab of my Web Site document in the Directory, and restart HTTP. When HTTP is loading, it will print out "HTTP Server: DSAPI DSAPIDLL Loaded successfully" (or whatever message you put in there), so you know it's working.

Now comes the task of actually writing the filter. Considering that I haven't written any C or C++ since college, I expect lots of fun with buffer overflows and mysterious crashes. Exciting!

Because I am insane, I decided to install Domino on my newly-minted 768 slice from Slicehost. That's just above the 512MB RAM minimum that Domino demands and half the recommended 1.5GB, so we'll see how it goes.

The Linux installation I went with for my slice was Ubuntu 10.10 64-bit, which firmly puts it into the realm of unsupported, as far as Domino is concerned. Fortunately, three pages I found made the process relatively painless:

1. http://www.collaborationmatters.com/blog/cmblog.nsf/dx/installing-lotus-domino-8.5.2-on-ubuntu-server?opendocument&comments
2. http://www.danilodellaquila.com/blog/how-to-install-lotus-domino-8.5-on-ubuntu-part-ii
3. http://stackoverflow.com/questions/3747789/how-to-install-sun-jdk-on-ubuntu-10-10-maverick-meerkat

The first wrinkle you run across is that Domino adores Java, but stock Linux installs do not. Fortunately, link #3 there explains how to cleanly install Java on your server. Namely, add this line to /etc/apt/sources.list:

deb http://archive.canonical.com/ubuntu maverick partner

Once you do that, you can run apt-get install sun-java6-jre, wince at the huge number of dependencies it requires, and let it do its thing.

After that, you're almost ready to start installing. If you're running on a 64-bit installation, you'll need the standard 32-bit libraries, since I don't think there's yet a 64-bit Domino server for any-old-Linux. That one's easy, though: apt-get install ia32-libs and you're all set.

The Domino installation proper was relatively straightforward. I created a "notes" user and associated group beforehand, picked and created some directories for the program files and data owned by that user, and went through its command-line wizard via ./install -console . For the final step, I told it to expect a remote installation and fired up the Remote Server Setup tool on Windows, which worked flawlessly.

The next step was the FP2 installer. As is the case with the Windows version, the installer for the fix packs is totally different than the normal installer for some reason. It initially complained about the size of my Terminal window and, rather than trying to convince it that my GUI terminal was plenty good enough for its installer, I took its advice and did export LOTUS_NOROWCOLCHECK=1 . After that, it complained about needing to know where the data directory was, but provided a similar instruction for setting that variable. Once that was in order, the installation went smoothly.

The next tough part was how to get Domino to run at startup. I've been familiar from time to time with various *nixes and their service setups, but it seems to change every couple of years, so I wasn't sure where to go with this. Fortunately, link #2 above did all the work for me. Since I found it long after installation, I didn't do the "customized distribution" bit, but I DID grab the Domino init script. Once I put that in place, modified the variables to point to my directories and user, and followed the other instructions, I was all set. Now I can control Domino via "service domino (start|stop|restart)" like you'd expect.

Due to the nature of Linux, the question of where to keep the program and data directories is a weird one. The defaults are something like "/opt/ibm/lotus" and "/local/notesdata", respectively, which would certainly WORK, but don't fit in with much else. I've fiddled with my placement a bit and settled on the same thing for the former, but "/var/lib/domino/data" for the latter. That allowed me to set up "/var/lib/domino/daos" and "/var/lib/domino/logdir" (for transaction logging) and keep things relatively clean. One thing to note: if you switch around the directories after the fact, make sure to change both the init script and the notes.ini in your program directory to reflect the new locations.

All told, it seems to be working pretty well. When you're using it like a normal Notes server, there's not much distinction other than the lack of OLE. It has all the same bugs (like the aggravating XPages Java classloader "X is not compatible with X" errors I run into constantly) and same features, which is really the idea.

Way back when I wrote my own blog back-end, I went out of my way to make decent-looking URLs that didn't betray the use of PHP for the code. It's not that using PHP was inherently bad, but the ".php" in the URLs was ugly and would have made it tough to move to any other back-ends. It's just one of those good-idea cleanliness things. Back then, I did it with Apache MultiViews and just had names like "archive.php", so I could make URLs like "/archive/2002/12" for that month's posts.

Since I've started using Domino, however, it's gotten a bit tougher. The parts of the URL within a Notes database can actually be pretty great - something like "/People/Foo+Fooson" is about as good as you'd want. However, it's the database path that kills me. You can set up URL substitution rules, but that's a weird combination of setting up a Directory Web Rule plus making sure that every place you make a URL in your code uses that name. For something like a view column, that'd mean either hard-coding your server-specific setting or sucking it up and using @WebDbName. For the most part, I've given up and gone with the latter, for the sake of portability.

XPages bring good news and bad news.

The good news: your application almost never needs to care about the database file path, ".nsf" extensions, or anything of the like. There may be edge cases where you do, but for the most part you can just start your URLs with "/" and the database path is filled in for you (at the cost of starting non-DB paths with "/.ibmxspres/domino").

The bad news: even though the database path reference is now handled by the runtime and not the programmer's code, I don't know of a way to override the behavior. I'm sure it's there SOMEWHERE, buried in the hordes of Java classes and method calls spawned for each page request, but I don't see it. What I'd really want is to tell the XPages runtime that, while "/wow/forums.nsf" is the "right" way to reference the database, I'd rather it just start with something like "/forums". It seems like something that must be handled in the xsp.properties file, but I haven't found anything about it yet. Admittedly, it's tough to search for - most pages with the keywords I want are talking about @WebDbName replacements in SSJS and how to references external Domino resources. If it's not present already, maybe it's something they'll add in a future point release. One can hope.