Continous Integration: Hudson jobs disappear on restart of Tomcat

Something peculiar happened today while we were playing around with newly installed continuous integration tool, Hudson. As I was very new to using Hudson, I was for a moment taken aback with what happened.  The story goes like this. A few days back, we decided to use Hudson as our CI tool. As a result, we could deploy the hudson.war and configure it to run our projects in no time. Everything seemed perfectly ok. We were all very much impressed with the awesome user interface, that Hudson provided for the configuration of our project.

We left the CI tool run for couple of days as a trial run and once we realized that everything is working fine as expected, we decided to go ahead with enabling of authentication/security settings for the users. Idea is, only an admin should be able to make changes to the CI and rest of the whole team was given some minimum privileges to view the running jobs and probably to do some force builds if required. Then we decided to restart the tomcat server in which Hudson is deployed in order to ensure that all the configuration changes done for security are applied and reflected.

Since the tomcat server was in a remote machine, we logged in via SSH as one of the regular users and restarted the tomcat. When we came back to the Hudson home page, we were greeted with a biggest shock of our life. All the projects/jobs that we configured previously was no more seen. They just disappeared! Neither seemed to have the user access controls we defined been applied. It was like a fresh deployment of hudson.war. We had no clue of what exactly had happened.

After little bit digging into the nightmarish problem in hand, we realized that actually when Hudson was deployed for the first time, it was done through the ‘sudo’ privileges of the remote machine. So Hudson persisted all the job details that we created under /root/.hudson/jobs folder.

But this time, when we logged in, we logged in as different user and were not using ‘sudo’ privileges to restart tomcat since we were able to restart the tomcat instance using this user account itself. As a result, Hudson simply tried looking for the previous configuration details at /home/user/.hudson/jobs folder while initializing, instead of looking at /root/.hudson/jobs. Since nothing was found there, it had nothing to initialize with and hence just simply showed itself as a fresh copy/as a newly installed version.

The solution to get back the disappeared jobs was very simple. This time we restarted the tomcat via ‘sudo‘ and everything (included our defined user access controls) came back in Hudson. Only when saw all jobs and configuration details are intact, we got a huge sigh of relief. What a stupidity isn’t it? Still I thought its worth sharing this information with you folks anyways. 🙂

So the moral of the story is, next time when you restart your tomcat and see nothing on Hudson dashboard, just check if you have all the configured jobs under your ~/.hudson/ folder.

Advertisements

About this entry