learning thread - recovery from kernel panic in debian 8.2

Discussion in 'all things UNIX' started by Palancar, Oct 18, 2015.

  1. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,594
    First time to hit this head on. Yesterday I updated two of my debian hosts and they took the 10-9-15 kernel update. I boot these hosts via usb sticks containing /boot and of course the loader. Everything went fine and I surfed for several hours without any issues noticed. Today I booted one of them and everything was perfect, slick, fast, etc... However; the other one (with the exact same update and kernel) doesn't boot and indicates a kernel panic. I did not have occasion to boot either of these until today. I read around about kernel panic and decided to grab my spare /boot flash (I created a perfect spare (even UUID's match) for each and keep them in storage) and the OS fired right up. The stored/spare flash is using a kernel updated from late August and its two behind the current one. The OS runs perfectly. So, now the dilemma is how to correct the problem and return my spare usb /boot to storage. I don't want to update my spare usb because its perfectly sync'd to a sector based clone of my entire linux partition just in case I ever need to do a full restore.

    Looking for suggestions as a learning tool. It would be easy for me to simply do the full sector restore on usb3 hardware its under an hour. Problem is that doesn't teach me anything and I am trying to learn linux a little bit more each day.

    fyi - once I had the OS up and running I did a sudo grub-install /dev/sdb on the /boot flash. The usb stick light was flashing indicating a smooth write process and the linux terminal indicated grub was written with no problems noticed. I thought I had this licked but upon reboot it still shows a kernel panic with lots of errors. Now that I think of it though I bet the grub install was still on the older kernel because that is what uname -a in a terminal showed.

    I am not aware of a way to start debian on one kernel and then jump to the newest one without a complete restart of course.

    Just looking for some suggestions. Roll back the kernel and then reload it?

    Again, if its a full restore that will be super easy but I learn nothing.
     
  2. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,594
    Now that I think about it I remember my system "flinching/hanging" for a second or so during the update process yesterday. I think a file or lib just got hosed. When I update I click on an exectable shell script (pasted below to make my point) on the desktop. I have used this little "script" to update too many times to count on all my VM's and hosts. I can't imagine this is a contributing factor but I"ll paste here for reference, or to use if somebody wants it.


    #! /bin/sh
    sudo apt-get update && sudo apt-get dist-upgrade && sudo apt-get autoremove
    read -p "*** press Enter to close terminal ***" nothing


    Thank all for your consideration of this circumstance. In the end it'll likely be my friend Mr. Murphy just came for a visit. LOL!
     
  3. amarildojr

    amarildojr Registered Member

    Joined:
    Aug 8, 2013
    Posts:
    1,985
    Location:
    Brasil
    Interesting how the latest Kernels could just go BOOOM for a lot of people. Even linux-grsec borked and is making boot impossible, and guess what? I don't have a backup of the working Kernel :argh:

    IMO, Stable breaks the least. Sure, you have quite an outdated system (if you don't use backports), but at least they work 99.9% of times. The con of having Debian Testing or Sid installed is that you don't know what the next updates will bring.

    Do you have apt-listbugs installed?

    Have you tried compiling a different Kernel directly from kernel.org?
     
  4. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,594
    No, I don't have those installed. I am using Debian stable for both OS's. Strange thing is one is working like a swiss watch and the other is pulling kernel panic. I find that interesting since they are exactly the same OS receiving the same kernel update. Also, "funny" is that I left Ubuntu due to daily updates to avoid such instances and I have never seen a kernel panic on any of my Ubuntu systems. I am of course still staying with Debian for a host of reasons.

    It is clearly the kernel because as I mentioned above; if I mount the troubled system using an older kernel (remember my backup usb /boot) it also runs like a swiss watch. The confusion comes for me because the one system is fine and the other is not. Hmmmm?

    Since I have a sector image backup I'll likely just blow it back on and see if it happens again when I take the same updates. I have a hunch it won't repeat itself, so stay tuned. I'll execute the simple restore tonight while I am watching TV.
     
  5. amarildojr

    amarildojr Registered Member

    Joined:
    Aug 8, 2013
    Posts:
    1,985
    Location:
    Brasil
    How are you getting those updates?
     
  6. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,594
    See second post in thread for executable shell script:

    http://ftp.us.debian""""""""""""" via VPN hops.
     
  7. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,594
    GOT IT ------------- > not a Debian issue!

    It took me a couple of trial and errors but in the end my /boot usb got hosed somehow during the update. I came home early to attend to this. I restored the linux OS partition using my sector image and it started fine with my "good" flash. That eliminated trash in the OS since the image was almost perfectly clean. I decided to do a grub-install from my linux OS to the "bad" usb. It didn't work properly. So, I decided to strip that flash completely down. Gparted it all to oblivion and started over. New partition table, ext2 file system (setting the correct UUID in a terminal), even manually writing the saved loader (446 bytes). Then I reset the grubloader/flash using the rescue menu on the Deb install DVR, just to be sure. Went great!

    Its working perfectly since I completely rebuilt the boot flash. After doing alot of reading I have run into numerous threads about kernel panic and suspect hardware condition. I will be watching that flash closely. I am giving it one chance assuming a fluke writing error. If it does it again its gone! I have a backup flash and my saved images make a restore super simple.


    IN THE END ----- I prefer to have discovered this rather than Debian causing the issues. I switched to Debian stable to avoid these types of issues. I learned something going through this process too. Never quite figured out why the system was screaming kernel panic, but now I have some experience with this. Strangely, it was fun at the same time as frustrating.
     
Loading...