If the dev directory is missing, does creating it make any difference? On my Ubuntu 8.04 system, /dev is of type tmpfs:

udev on /dev type tmpfs (rw,mode=0755)

So I would think that the contents of /dev don't survive a reboot, so maybe just having the directory there to serve as a mount point is enough.

Alternately, Ubuntu has a /sbin/MAKEDEV shell script (at least, over here it does), which could be used to fill up an empty /dev if you were so inclined. I think you cd to the new dev directory, and then run 'MAKEDEV generic'.

I'm unable to cd into the dev folder :)

inspection of my daemon.log shows this about the time of the crash

Jun 30 05:42:05 doughnut modprobe: FATAL: Could not load /lib/modules/2.6.18.8-domU-linode7/modules.dep: No such file or directory

Jun 30 05:42:05 doughnut modprobe: FATAL: Could not load /lib/modules/2.6.18.8-domU-linode7/modules.dep: No such file or directory

Jun 30 05:48:58 doughnut init: tty1 main process (2062) killed by TERM signal

Jun 30 05:48:58 doughnut mysqld[29849]: 080630 5:48:58 [Note] /usr/sbin/mysqld:

Normal shutdown

Jun 30 05:48:58 doughnut mysqld[29849]:

Jun 30 05:48:58 doughnut mysqld[29849]: 080630 5:48:58 InnoDB: Starting shutdown…

Jun 30 05:49:00 doughnut mysqld[29849]: 080630 5:49:00 InnoDB: Shutdown completed; log sequence number 0 4947393

udev and modprobe are a mystery to me made even more difficult by having the server on a VPS.

Was your /etc/fstab changed in some way so that /dev is no longer mounted?

@gsf:

Was your /etc/fstab changed in some way so that /dev is no longer mounted?
I don't think fstab has anything to do with that, though. I've checked on 3 different machines now, and neither references /dev in the /etc/fstab file:

One of my Linodes (Debian Linux):````
/dev/ubda / ext3 defaults 0 0
/dev/ubdb swap swap defaults 0 0
proc /proc proc defaults 0 0

The firewall computer at work (also Debian Linux, not virtualized in any way):````
proc            /proc           proc    defaults        0       0
/dev/hda1       /               ext2    defaults,errors=remount-ro 0       1
/dev/hda5       none            swap    sw              0       0
/dev/hdd        /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

The primary e-mail server at work (Gentoo Linux, also not virtualized in any way): /dev/hda1 /boot ext2 defaults,noatime 1 2 /dev/hda2 none swap sw 0 0 /dev/hda3 / ext3 noatime 0 1 /dev/hdb1 /brukere ext3 noatime 0 1 /dev/hdb2 /home/vmail ext3 noatime 0 1 /dev/cdroms/cdrom0 /mnt/cdrom auto noauto,ro,user 0 0 proc /proc proc defaults 0 0 none /dev/shm tmpfs nodev,nosuid,noexec 0 0
For now, I think we can eliminate /etc/fstab as being the "culprit" in this situation.

/proc /proc proc defaults 0 0

/sys /sys sysfs defaults 0 0

/dev/xvda / ext3 defaults 0 1

/dev/xvdb none swap defaults 0 0

I don't think it to do with fstab, but thanks for the help.

It would appear as if udev isn't being called on start up or perhaps its not being called early enough.

@jimflip:

I'm unable to cd into the dev folder :)

With your disk image mounted under Finnix, you should be able to cd into the dev folder - it normally contains a default set of device nodes. udev mounts a tempfs on /dev at boot, making the defaults invisible and replacing them with the device nodes it has created in the tempfs. Check /dev permissions:````
drwxr-xr-x 4 root root 12288 Jul 4 09:03 dev

If necessary, remove and recreate /dev I shouldn't matter that the defaults are missing since udev will override them anyway.

Check the contents of /etc/udev/rules.d. It should look something like this:````
05-options.rules                  70-persistent-cd.rules
05-udev-early.rules               70-persistent-net.rules
20-names.rules                    75-cd-aliases-generator.rules
30-cdrom_id.rules                 75-persistent-net-generator.rules
40-basic-permissions.rules        80-programs.rules
40-permissions.rules              85-hdparm.rules
45-fuse.rules                     85-hwclock.rules
60-persistent-input.rules         85-ifupdown.rules
60-persistent-storage.rules       85-pcmcia.rules
60-persistent-storage-tape.rules  90-modprobe.rules
60-symlinks.rules                 95-udev-late.rules
61-persistent-storage-edd.rules   README

Check that /etc/udev/udev.conf is not corrupted. Contents on a fresh install:````

udev.conf

The initial syslog(3) priority: "err", "info", "debug" or its

numerical equivalent. For runtime debugging, the daemons internal

state can be changed with: "udevcontrol log_priority=".

udev_log="err"
````

Thanks all for your help, it's working again now, which is a big relief.

I created a new /dev folder and ran 'MAKEDEV generic' in it. Did a reboot and bingo…it works.

Udev seems a bit fragile to me and found lots of people having problems with it on the internet..it does all seem a bit odd, I will investigate it further.

Again many thanks all.

I know you had problems with it… but the truth is that udev is pretty standard at this point.

Perhaps some rogue app deleted your "base" dev nodes and caused things to break? It seems like the deeper problem is that your /dev/ got cleared out.

> Perhaps some rogue app deleted your "base" dev nodes and caused things to break? It seems like the deeper problem is that your /dev/ got cleared out.

I agree, this all happened after I did a apt-get upgrade followed by a manual reboot.

My server is 7.10 server upgraded to 8.04 server Ubuntu, I have nothing too funky installed and everything is from standard apt repositiories.

I have had problems previously with udev on doing upgrades with ubuntu on a desktop machine.