How do I solve Login Problem

Yesterday I tried to login to my Linode server (Ubuntu 18.04), but I got an access denied response from the terminal. The website (Drupal 7) was working fine. I then loggedin to my Linode account and powered down the server in order to change the master password. After starting the server again, the new password did not work and the Drupal 7 website no longer worked. I checked the LISH terminal and saw a repeating "failed to mount error". I then rebooted to "Restore" mode and checked the file system. There were no errors. I then tried to manually change the password while in Restore mode but once I rebooted. I again could not login to the server (or the website). Linode support has provided guidance but at this point, I've tried the two things they suggested. Since I am unable to login to the server, I may need to rebuild the site. I'm reaching out to the community to ask for any advice prior to taking this step.

Below are some of the error messages the Linode support saw when they checked the server.

[FAILED] Failed to mount Kernel Deb[ 9.760454] systemd[432]: ufw.service: Failed to execute command: No such file or directory
ug File System.
See 'systemctl status sys-kernel-debug.mount' for details.
[FAILED] Failed to mount POSIX Message Queue File System.
See 'systemctl status dev-mqueue.mount' for details.
[FAILED] Failed to mount Huge Pages File System.
See 'systemctl status dev-hugepages.mount' for details.
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[FAILED] Failed to start udev Coldplug all Devices.
See 'systemctl status systemd-udev-trigger.service' for details.
[FAILED] Failed to start Create list of requ…vice nodes for the current kernel.
See 'systemctl status kmod-static-nodes.service' for details.
[FAILED] Failed to start Uncomplicated firewall.
See 'systemctl status ufw.service' for details.
Starting Create Static Device Nodes in /dev…
[ 9.796756] systemd[438]: systemd-tmpfiles-setup-dev.service: Failed to execute command: No such file or directory
Starting File System Quota Check…
Starting Load/Save Random Seed…
[FAILED] Failed to start Create Static Device Nodes in /dev.
See 'systemctl status systemd-tmpfiles-setup-dev.service' for details.

Any help or advice would be greatly appreciated.

Dave

8 Replies

In discussions with Linode support staff, here's what we've figured out: I am unable to log in to the server because the OpenDKIM Domain Keys service is not loading during the server boot process. In order to see why that is the case, I would normally use systemctl status to get the details for this failure to load. However, I can't log in to the server so I'm stuck. Is there anyway that I could do this in Rescue mode? Or is there a way for me to disable this service in Rescue mode? If the answer to both of these is no, then I presume that the only course of action is to transfer the data and rebuild the server. Again, thank you for providing all of your guidance.

There is a way to disable the service in rescue mode.

  1. Boot to rescue mode. Make sure /dev/sda is mounted as a regular disc to the rescue mode system (I don't use Linux anymore so I'm unsure of the details…maybe someone else can fill those in).

  2. Find the file called opendkim.service (it's in <rescue-mode-mountpoint>/lib somewhere I think) and rename it to opendkim.service.orig.

  3. Find the config file for your mail server and disable it from talking to the opendkim milter. Since I don't know which mail server you use or how it's set up, I can't provide more details here.

  4. Exit rescue mode and try booting your system again. Rinse, wash, repeat…

I had exactly this same problem…with exactly the same culprit (a systemd f'up). I could never get the installed package to work correctly ever again (another systemd f'up). I finally got back on the air by building opendkim from source.

This was the last straw in a series of lost weekends due to systemd. I've sworn off Linux forever.

-- sw

Thank you @stevewi for your response and suggestion. I'm not a server person much less a Linux fan. I certainly don't want to burden you with my problem, either. At a minimum, I just want to get the site to boot up and make it possible for me to log in again. Can I bother you for a follow up query?

For Point 2 of your solution, I found three instances of opendkim.service, as follows:

/media/sda/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/opendkim.service
/media/sda/lib/systemd/system/opendkim.service
/media/sda/etc/systemd/system/multi-user.target.wants/opendkim.service

Should I add the .orig extension to all three of them?

I'm using postfix as my mail server. For Point 3 I searched the following postfix config files for something related to opendkim milter but found nothing.

/media/sda/etc/postfix/main.cf
/media/sda/etc/postfix/master.cf
/media/sda/etc/postfix/main.cf.proto
/media/sda/etc/postfix/master.cf.proto

There's also a configuration file in /etc/opendkim. What should I be looking for in these config files?

I can't thank you enough for sharing your experience. I'll keep looking for solutions but you've given me hope I may see the website again.

Dave

For Point 2 of your solution, I found three instances of opendkim.service, as follows:

  • /media/sda/var/lib/systemd/deb-systemd-helper-enabled/multiuser.target.wants/opendkim.service
  • /media/sda/lib/systemd/system/opendkim.service
  • /media/sda/etc/systemd/system/multiuser.target.wants/opendkim.service
     
    Should I add the .orig extension to all three of them?

[emphasis added]

Yes. What you are doing here is preventing systemd from trying to start opendkim. I'm pretty sure the 2nd one is the actual file and the other two are symlinks. When the system starts up, you're going to rename them back and then remove opendkim (which should remove all these files/links)

I'm using postfix as my mail server. For Point 3 I searched the following postfix config files for something related to opendkim milter but found nothing.

  • /media/sda/etc/postfix/main.cf
  • /media/sda/etc/postfix/master.cf
  • /media/sda/etc/postfix/main.cf.proto
  • /media/sda/etc/postfix/master.cf.proto

opendkim operates as a milter (mail filter). That's how postfix & opendkim communicate. In the file /media/sda/etc/postfix/main.cf, you will see something like the following (yours will look different):

smtpd_milters =
  unix:/var/run/pyspf-milter/pyspf-milter.sock,
  unix:/var/run/opendmarc/opendmarc.sock,
  unix:/var/run/milteropendkim/opendkim.sock

This is from my postfix config. It could also look like this:

smtpd_milters =
  localhost:1234,
  unix:/var/run/opendmarc/opendmarc.sock,
  localhost:5678

Obviously, I made up the numbers. The numbers are TCP port numbers. I'm just trying to give you examples of anything you might see.

You want to comment all that out (put a # in front of each line)…the whole setting for smtpd_milters. The way postfix configuration is written is:

parameter = foo, bar, baz

or

parameter = 
   foo,
   bar,
   baz

You specify multiple values by separating them with commas. The last value of a set of multiple values is not followed by a comma (newlines don't matter when postfix is processing multi-valued parameters).

You want it to look like this:

#parameter = foo, bar, baz

or

#parameter = 
#   foo,
#   bar,
#   baz

 

There's also a configuration file in /etc/opendkim. What should I be looking for in these config files?

That can stay. That's the config for opendkim. If opendkim doesn't start, it won't get read.

I hope this helps.

-- sw

Thanks again. I followed your instructions but it appears there are other issues besides opendkim not loading. The server is not fully operational and there are other error messages in the LISH console. In the meantime, Linode has restricted the server because of two DOS from the server. I guess I need to rebuild the site. The mysql database backup I have is not current so can anyone tell me how to make a backup of the database when in Rescue mode. I really could use some expertise.

The mysql database backup I have is not current so can anyone tell me how to make a backup of the database when in Rescue mode. I really could use some expertise.

Without mysqld running (which will definitely be the case in rescue mode), I doubt you'll be able to do a dump. If you knew where the database itself was stored, you could save the files and try to use them on your new site. Sorry, I don't know enough about mysql to be able to tell you the location. Also, I don't use Linux…

there are other error messages in the LISH console.

What are these? These may be solvable.

-- sw

P.S. If you rebuild the site, you should do it on Ubuntu 22.04…the current LTS release.

It's okay. I found Linode documentation to do this. With multiple error messages and network restrictions due to DOS, I'm just going to rebuild the server. Thanks all!

I really really appreciate your responding and helping me. As I said, I'm not a qualified Linux user so my best bet is to just start over from scratch and rebuild the server. Yes, I'll use 22.04. Again, thank you so much for your suggestions. It's always nice to know others are willing to share their expertise.

Dave

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct