When Ubuntu Server doesn't let you keep the internet device drivers

So I ran into a problem while setting up Ubuntu Server 16.04.1 LTS for my cluster: one of my computers had restricted hardware drivers for the wifi and ethernet devices, and while the drivers were available on the install media, they weren't transferred over.  Something about non-free stuff.  So after installation, I was stuck with a computer that had no access to the web.

After a ton of searching, I a) found what the hardware problem was, and b) realized that I can use the install media as the installation source - that might sound obvious, but the keyword there is "can", not "should be able to". It's badly documented and very not-obvious.

Anyway, here's the source for using the install media.  Part of the install process is choosing which sets of packages you want to include - things like OpenSSH Server stuff or Mail Server stuff.  There's also one called Manual Package Selection.  It didn't seem to do anything during the install (though I did walk away, and it might have timed out), but you can reenter the tool after the installation is finished and you've rebooted into the new OS.

After rebooting, log in and plug the ubuntu server install media into the computer.  Run:

sudo tasksel install manual

This command will find the install media, and 'install' the manual package selection 'package'.  In other cases, it would actually install stuff - in this case, it just gives you a shell prompt.  At this prompt, you somehow have full internet access - I don't know how it happened.  It was automagical.  In my case, the next step was editing the sources list to enable the universe repository (already seemed enabled - automagic of the tasksel command?).  Source on the actual internet driver fix.

sudo nano /etc/apt/sources.list

And a line that looks like this

deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted

Should be changed to look like this:

deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted universe

[ctrl]-x to leave.  Then, run 

sudo apt-get install r8168-dkms

And the driver is installed.  Reboot and the internet should work.

Install a package without internet

So first of all, this isn't original.  Credit goes here.  But it's fantastic, and I wish I'd known about this a long time ago.  As usual, for my own memory/use: and actually, I'm just going to clean up what the other guy said.  He did a great job.

On the Internet-less computer:

In the terminal enter:

PACKAGENAME=<The name of the Package to install>

and then

apt-get -qqs install $PACKAGENAME | grep Inst | awk '{print $2}' | xargs apt-cache show | grep 'Filename: ' | awk '{print $2}' | while read filepath; do echo "wget \"http://archive.ubuntu.com/ubuntu/${filepath}\""; done >downloader.sh

A ready-to-use downloader for the package has now been created in the home folder.  Open your home directory in the file browser and move the file downloader.sh to the top-level directory of your flash drive.  Then eject your flash drive.

On the computer with Internet:

Insert your flash drive, and open your flash drive in the file browser.  Copy the location of your flash drive:

[CTRL]-L

[CTRL] C 

Move into the directory of the flash drive.  In a terminal this time, type:

cd [CTRL]+[SHIFT]+V 

Run the downloader:

bash ./downloader.sh

Wait for the download to complete and eject your flash drive.

Back to the Internet-less computer:

Open your flash drive in the file browser.  In the browser, type the following to copy the file location of the flash drive.

[CTRL]-L

[CTRL]-C

Move into the directory of the flash drive.  In a terminal this time, type:

cd [CTRL]+[SHIFT]+V 
sudo dpkg --install *.deb

That's it!

A Homemade Beowulf Cluster: Part 1, Hardware Assembly

A beowulf cluster lets me tie miscellaneous computers together and use their cpus like one large processor...I think.  Never done this before, still working on the details.

I'm building this with random laptops: generally i5s, and I think one's a Core Duo - it's half decent thing from 2011.  Might even throw in an RPi2 for good measure.

Make sure you read through this before starting.  You want to know what you're getting into.  Watch out, though - this is a long post.

Notes

Since we're working on multiple computers here, not everything is going to be cut-and-paste.  I will make sure that it's as clear as possible, however.  There won't be any hand-waving or assumptions of prior knowledge.

I'm doing this with Linux Mint 18 on my primary laptop.

Primary Sources

Setting up the cluster: src 1,  src 2, src 3

Hardware Ingredients

• Since the benefit of this tool is sharing computations between computers, you need a way to route that information.  Hence, an ethernet switch.  Go for a gigabit, since you don't want the switch to be your bottleneck.  I was cheap and got myself the 5-port version, and I'm already kicking myself.  Go for the 8-port version at least.  Here's the 5-port version I got, for consistency's sake. 

• A ton of ethernet cables.  You can do with short ones, but you'll want to connect the switch to your router so you can access the computers from outside their own tiny network.

• Leftover computers.  You won't be using these for anything else, so make sure you don't need them. 

Computer Preparation

Install Ubuntu Server to each computer. I used 16.04.1 LTS.  For those with a penchant for funny names (or who need to deal with annoying, obtuse colloquialisms on the web), that's the Xenial Xerus edition.

You'll need to keep track of computer names ("hostnames"), and install consistent users (so the username and password are the same on each machine).  I like to increment the computer names with letters.  For the purposes of this guide, we'll use:

hostname:    grendel-a
username:    beowulf
password:    hrunting

Create installation USB flash drive

Go here to download the 16.04.1 LTS server ISO.  That link initiates the download; here's a bit of context.

Burn the ISO to a USB flash drive. Note that you'll lose everything on the USB you use.  Run the following command twice, once before plugging in your USB and then a few seconds after.  The new entry when you run it the second time is your flash drive.

lsblk

For example, this is the output when I run that command with a flash drive plugged in.  Note on the right where it specifies the mount point (at /media/me/storage), and on the left where it shows me the name is sdb.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 167.7G  0 disk 
├─sda1        8:1    0   487M  0 part /boot
└─sda5        8:5    0 167.2G  0 part 
  ├─mint--vg-root
  │         252:0    0 159.3G  0 lvm  /
  └─mint--vg-swap_1
            252:1    0   7.9G  0 lvm  [SWAP]
sdb           8:16   1   1.9G  0 disk 
└─sdb1        8:17   1   1.9M  0 part /media/me/storage

If your USB is mounted, it has to be unmounted first - else weird things can happen in the next step.  Trust me: once I didn't unmount a partition before copying it with dd, and my MBR was wiped out instead.  In the example above, unmounting would work like this:

umount "/media/me/storage"

If we pretend your USB is the sdb device, this is the command you'd run (I'm assuming the ISO was saved to the default "~/Downloads" location).  Swap out the 'sdb' part with what the lsblk command indicated.  Also be aware that if you mess this up, you will probably destroy whatever computer you're running the command with.  Just FYI.

sudo dd if=~/Downloads/ubuntu-16.04.1-server-amd64.iso of=/dev/sdb bs=4M status=progress

Install to each computer

You're gonna have to follow this procedure with each computer in the cluster.  There isn't a simple way around it, that I'm aware of.

Plug in the flash drive, and reboot/turn on the computer.  Make sure it's connected to the web.  Press ESC, F1, F2, F11 or F12 to choose a startup device...if those don't work, Google:

"how to choose startup device BIOS" & [your computer type]

e.g., "ThinkPad T420".

When it boots up, there's a few settings to make sure of.  Most of them should be straightforward: choosing a keyboard layout and a default language, for instance.  Just in case of problems, here's a full walkthrough.

All you want is a simple installation.

Don't bother with detecting the layout...if you're in the US or have an english keyboard, you can just stick with the defaults.  Worst case, start over.  There's a few more images after that one to do keyboard stuff, but you get the idea.

grendel-a, grendel-b, etc.  Makes it easy to keep the computers straight. After you've entered the hostname as above, press [enter].

I'm using the same thing for the full name and the username.  Makes it simple. Press [enter].

Since the username is the same as the full name, just press [enter].

That's the standard password for the computers.  I used the [down arrow] and pressed [space] to select that radio button and show the password.  Just press [enter].  It'll ask you to reenter the password for verification; do so, and press [enter] again.

Don't bother with encryption; this is a cluster designed for speed, and that just makes disk access slower.  For the next screen, check your time zone.  If it's wrong, the adjustment screen is pretty intuitive.

Press [enter] here to go with the default option.  You want to use the entire disk, and LVM could be useful (it's also an incredible pain).  Press [enter] again on the next screen after verifying the destination for the install.  It's probably the largest drive - the small one is probably your flash drive.

It asks for confirmation; [right arrow] and [enter].

I like to use a standard size for the installation that leaves lots of room to spare for other things on the drives.  Might be able to do some kind of shared network storage for the computations with the rest.  If it lets you, go for 50.0 GB.   [enter].

Another confirmation.  [right arrow] + [enter].

Unless you have weird internet, leave this blank and press [enter].  If you do have a weird internet setting, then I can't help you.

I skipped the auto updates, since they won't positively impact the performance of the cluster, and might slow it down at times.

Ok, this is the one that really matters.  Addendum: also select 'Manual Package Selection'.  For some machines with obscure drivers, this seems to ensure the drivers get installed.  You need the OpenSSH server in order to communicate with your cluster via the terminal. (at least, I'm pretty sure you do.  I didn't want to spend the time to find out for sure.)  [arrow down] to it, then [space] to select.  [enter] to move on.

Yes; you want GRUB installed to the MBR.  [enter].

...and that's it.  Reboot, remove the flash drive, and the new OS is installed.

Final Adjustment: power management

There is one more thing, though: you'll be running a bunch of these computers in the cluster, and you don't want to have to deal with them individually.  For one thing, that would be a huge mess.  So to keep them consolidated, you'll want to keep their lids shut while they're running: and if you don't change a setting, they'll just go to sleep when you close the lid.  src. 

sudo nano /etc/systemd/logind.conf

Find the line:

#HandleLidSwitch=suspend

and change it to:

HandleLidSwitch=ignore

Then restart that bit of the system:

sudo service systemd-logind restart

Assembling the Cluster Hardware

This is pretty straightforward: start with your ethernet switch.

1. Run a cable from your router to the #1 port on the switch.
2. Run a cable from each computer in the cluster to a port on the switch.  I'm not going to try and correlate port numbers and computer numbers.  Pretty sure it doesn't matter.

And that's all it takes to assemble the cluster hardware!  You should be able to ping your computers and get a response:

ping grendel-a

If you don't get replies, you have a problem.  Google is your friend.  Worked for me.

A Recurrent Neural Network to generate sentences: step-by-step assembly (Mint version)

This one predicts (ok, procedurally generates) text word-by-word.  Others do it letter-by-letter, but then you have to deal with misspellings.  src. Inspiration.

Dependencies

I'm basing this off an installation script referenced in the github readme (here).  Instead of blindly running the script, I checked its contents and discovered it fails when run on linux mint.  So I simplified it, and changed/added a thing or two since it wasn't working anyway.  In case anyone wants to blindly run my version, here it is all cleaned up.

Updating your sources is always a good idea prior to installing a bunch of software.  Also, the package below makes it easy to add new repositories (see below).

sudo apt-get update
sudo apt-get install -y python-software-properties

Now things are getting serious.  This stuff includes GCC 4.9, since it seems GCC 5 isn't compatible with Torch 7.  There was a weird error where cmake wasn't installed when I put all the packages in the same apt-get install command - I split out the packages to their own lines, ran my install script again, and it worked.  Very odd.  Hence, my inefficient code below.

sudo apt-get install -y build-essential 
sudo apt-get install -y gcc 
sudo apt-get install -y g++ 
sudo apt-get install -y curl 
sudo apt-get install -y cmake 
sudo apt-get install -y libreadline-dev 
sudo apt-get install -y git-core 
sudo apt-get install -y libqt4-core 
sudo apt-get install -y libqt4-gui 
sudo apt-get install -y libqt4-dev 
sudo apt-get install -y libjpeg-dev 
sudo apt-get install -y libpng-dev 
sudo apt-get install -y ncurses-dev 
sudo apt-get install -y imagemagick 
sudo apt-get install -y libzmq3-dev 
sudo apt-get install -y gfortran 
sudo apt-get install -y unzip 
sudo apt-get install -y gnuplot 
sudo apt-get install -y gnuplot-x11 
sudo apt-get install -y ipython
sudo apt-get install -y libpcre3-dev

These are for a program called OpenBLAS, which stands for Basic Linear Algebra Subprograms.  Here's a bit more on the concept.

sudo apt-get install -y libopenblas-dev liblapack-dev

Install Torch (finally).  Note that when running the ./install.sh command, I'm letting bash automatically enter "Y" when prompted for a user response.  It's requesting permission to add torch to the system path in the .bashrc: essentially, creating a system shortcut for each initialization of the bash terminal.

git clone https://github.com/torch/distro.git ~/torch --recursive
cd ~/torch; 
echo "Y" | ./install.sh 
source ~/.bashrc

These tools are installed with Lua, a programming language that came with Torch.

/home/$USER/torch/install/bin/luarocks install nngraph
/home/$USER/torch/install/bin/luarocks install nninit 
/home/$USER/torch/install/bin/luarocks install optim
/home/$USER/torch/install/bin/luarocks install nn
/home/$USER/torch/install/bin/luarocks install underscore.lua --from=http://marcusirven.s3.amazonaws.com/rocks/
sudo /home/$USER/torch/install/bin/luarocks install lrexlib-pcre PCRE_DIR=/lib/x86_64-linux-gnu/ PCRE_LIBDIR=/lib/x86_64-linux-gnu/ 

Make sure to add Torch to your PATH.  This ensures that when you run the command to use Torch, your computer can actually find the program.

to_path="/home/$USER/torch/install/bin"
echo "PATH=$PATH:$to_path" >> /home/$USER/.bashrc
source ~/.bashrc

With that, all dependencies should be satisfied. Time to install.

Installation

Clone the software from github.

mkdir ~/projects
cd projects
git clone https://github.com/larspars/word-rnn.git

That's actually all there is to it.  Now cd into the word-rnn directory to run the test stuff.  Before the tests and tools, though, there's a fix that you have to perform.

Extra Fixes

I'm running torch on a gpu-less computer.  There's a glitch that occurs when running the test script in that scenario.  To avert it, you have to change the name of a function from CudaTensor to Tensor.  See here for details.  

cd word-rnn/util
nano SharedDropout.lua

The third line should look like this:

SharedDropout_noise = torch.CudaTensor()

Change it to this:

SharedDropout_noise = torch.Tensor()

And save.

CTRL + O
CTRL + X

Running word-rnn

Now you can run the test function.  Be aware that on a 4-core 2.8GHz i7 processor, this command took 21 hours to complete.  BUT it's a very cool command, and the result is amazing.

th train.lua -gpuid -1

To be continued...

LVM GUI Management

There's only one GUI tool for working with the Logical Volume Management paradigm, and in Fedora it's already been deprecated without a functional replacement.

(Apparently LVM is the partitioning scheme of the future)

Anyway, this is a personal note to myself that system-config-lvm is the best way to go, and it's available from the Fedora archives here.  To go straight to the download, go here.

It was last available in Fedora 19, so who knows how long it'll work for.

Installing Django

This is an incredible pain to figure out.  Dependencies - python 3.5, and don't forget to set up virtualenvs...whatever those are.  Don't get me wrong: I've done plenty of python, but never really wanted to deal with something that complicated.

The goal here is to get django installed.  Stretch goals, dealt with later, are a) to get django-based website running locally and b) to do lots of cool pythonic things in the background.

Based on further exploration of how these things work, I am amazed how complicated this is.  And the documentation is basically written for someone with complete conceptual understanding of the software, who just forgot the commands they need.  Not that beginner-friendly.

Sources

Installing django
virtualenv

virtualenvwrapper

Dependencies

Make sure python is installed. And the other stuff.

sudo apt-get install python3.5
sudo apt-get install python-pip
sudo pip install virtualenv

Create your virtualenv (src)

Apparently, it's good to store your virtualenvs all in the same place.  The recommended location is ./.virtualenvs, so that's what we'll use.

mkdir ~/.virtualenvs
python3 -m venv ~/.virtualenvs/myfirstdjangowebsite

Note: if you're using ubuntu, that last command might not work.  If it doesn't, you'll have to make sure the python3 versions of everything are installed and then use a different form of the command.

sudo apt-get install python3-pip
sudo pip3 install virtualenv
virtualenv --python=`which python3` ~/.virtualenvs/myfirstdjangowebsite

Now activate the virtualenv.  If the first command doesn't work, use the second.  Remember: you have to activate the virtualenv in every new terminal window.

source ~/.virtualenvs/myfirstdjangowebsite/bin/activate
. ~/.virtualenvs/myfirstdjangowebsite/bin/activate

Install Django

Once that virtualenv is set up, install django.

pip install Django 

Django is now available within the virtualenv you set up.

Raspberry Pi Zero USB Audio on Raspbian Jessie (method 2)

Well, this is the second way to skin the cat.  Note that arecord ignores what went into the recording section of the config file specified below.  See the execution section for actually making a record or playing something.

Followed the instructions of OP in this post, more or less (as detailed below).  Note the extra information provided by the top post at this link.

For some reason, the space on the RPi's SD card filled up completely after following those instructions and messing with the results.  Not even enough space to run ls on a big directory.  Something I'd expect if I'd run arecord instead of aplay, and left it going indefinitely.

Log into the pi.

ssh pi@raspberrypi

Install a dependency (don't worry, it's small).

sudo apt-get install libasound2-plugins

Edit the ALSA config file, but first backup the original.

cp ~/.asoundrc ~/.asoundrc.bak
sudo nano ~/.asoundrc

Add the following to the file.

pcm.!default {
              type asym
              playback.pcm "defaultplayback"
              capture.pcm "defaultrec"
                hint{   show on
                   description "default play and rec koko"
                }
      }

      pcm.defaultrec {
              type plug
              slave {
                  pcm "hw:1,0"
                  rate 48000 
                  channels 2
                  format S16_LE
              }
                hint{   show on
                   description "default rec koko"
                }

      }

  pcm.defaultplayback{
          type rate
          slave.pcm mix1
          slave.rate 48000 

          #Intel(R) Core(TM)2 Duo CPU     E7500  @ 2.93GHz:

            #converter "samplerate_best"     # perfect: 16%cpu, maybe overkill
            #converter "samplerate_medium"   # almost perfect: 6%cpu
            #converter "samplerate"          # good: 4%cpu, definitely usable
            #converter "samplerate_linear"   # bad: 2%cpu, way better than default wine resampler
            #converter "samplerate_order"    # very bad: 2%cpu, like the default wine resampler

          converter "samplerate"

            hint{   show on
                   description "default play koko"
            }
  }

    pcm.mix1 {
            type dmix
            ipc_key 1024
            slave {
                pcm "hw:1,0"
                rate 48000
                periods 128
                period_time 0
                period_size 1024 # must be power of 2
                buffer_size 65536

            }
    }

Execution

To play a file (of the proper format):

aplay file.wav

To record a file:

arecord -f cd file.wav

Read the aplay or arecord manual to see what's up with the -f cd thing.  Essentially, it's specifying the format the audio should be in.

man arecord