Tag Archives: linux

Linux: global, per-user and per-process limits on number of file descriptors

I have a problem: I am trying to test a demo server designed to deal with LOTS of simultaneous connections. And my tests fail at a small number because of socket.error: (24, ‘Too many open files’). I need to make the system let my test program open lots of sockets.

What are the current limits?

  • To obtain the current system-wide maximum number of file-descriptors-per-process, use 'cat /proc/sys/fs/file-max' or 'sysctl -a | grep fs.file-max'.
  • There is also a limit on how many file-descriptors a user may create: 'ulimit -n'

First we shall increase the system-wide limit: edit /etc/sysctl.conf and set the value of fs.file-max to the desired maximum. Run sysctl -p to effect the changes.

Now, let’s increase per-user limits. Edit /etc/security/limits.conf and put something like this at the bottom:

parijat hard nofile 400000

where ‘parijat’ is my login on this system. Logout and login again. ulimit -n should print 400000 now.

Advertisements

Fixing DNS servers in NetworkManager: Ubuntu Hardy


My office uses an ISP supplied DSL router to get its connectivity. The DSL router functions as a DHCP and DNS server for internal servers. However, its DNS is broken: it used to stop working every now and then, and it does not work at all now. We’ve setup an alternative DNS server, but there seems to be no way to tell the router to specify that DNS server’s address in the lease it gives out. (Well, we could set up an alternative DHCP server as well, I guess, but we haven’t gotten around to it.)

So, every time my desktop machine gets a new DHCP lease I have to go edit its /etc/resolv.conf to point to a working DNS server. This is painful.

This article describes two ways of solving the problem. I’ve just tried the first way: the jackol’s den » Fixing NetworkManager DNS issue in Ubuntu (Hardy Heron/Gutsy) – Mikhail Esteves.

  1. edit /etc/dhcp3/dhclient.conf
  2. Remove domain-name-servers from the request-options list.
  3. Uncomment “prepend domain-name-servers” line and edit it to use my DNS servers.

Let’s see.


Up and running with Emacs and Git

I code in Emacs. Emacs has good support for version control tool interaction in the vc-mode package. I want to use git and have vc-mode work with git. I also want access to special git commands that the vc-mode does not export directly. The notes below are for Ubuntu Hardy, which is what I use all the time.

Install Git

First I made sure I had git installed:

$ sudo aptitude install git-core

(I use aptitude instead of apt when I want ‘recommended’ packages to be installed automatically. Basically, I am letting aptitude choose and install other packages which are not strictly needed to install git-core, but which Ubuntu/Debian packagers think might be useful for me ;-)).

Create a template python package

$ cd workspace
$ paster create hello

I am using Python paste, paste script and paste deploy tools. The documentation for these sucks. Basically, I just use paste script to create the template for a project that can be packaged with setuptools. I am also doing this in a virtualenv. More on them some other time.

I end up with this structure:

$ cd ~/workspace/hello
$ tree
.
|-- build
|   `-- lib
|       `-- hello
|           `-- __init__.py
|-- hello
|   `-- __init__.py
|-- hello.egg-info
|   |-- PKG-INFO
|   |-- SOURCES.txt
|   |-- dependency_links.txt
|   |-- entry_points.txt
|   |-- not-zip-safe
|   `-- top_level.txt
|-- setup.cfg
`-- setup.py

5 directories, 10 files

Now I can build a tarball or egg with:

$ python setup.py bdist_egg
$ python setup.py bdist

Users can take the output and install this with ‘python setup.py install’.

Initial git commit

$ git init

$ git add -a .

$ git commit # write a suitable commit message

$ git log

$ git status

Emacs+git

On ubuntu, the git-core package contains emacs support files. I just add this to my .emacs
;; -----------------------------------------------------------------------------
;; Git support
;; -----------------------------------------------------------------------------
(load "/usr/share/doc/git-core/contrib/emacs/git.el")
(load "/usr/share/doc/git-core/contrib/emacs/git-blame.el")
(load "/usr/share/doc/git-core/contrib/emacs/vc-git.el")
(add-to-list 'vc-handled-backends 'GIT)

Now, basic vc-mode works:

C-x C-f ~/workspace/hello/setup.py
C-x v l # vc-print-log
C-x v g # vc-annotate

Direct git interaction

M-x git-status

git-status-mode is the major mode for interacting with git directly. Typing h in this mode bring up a menu of commands and keybindings.

Adding a file

I add two files: hello/speaker.py and hello/main.py.  Then:

M-x git-status

Navigate to speaker.py and main.py and type ‘a’ (for git-add-file) on both.

Type ‘c’ (for git-commit-file). Type in a log message and C-c C-c to finish the log message and commit.

Adding files to gitignore

I create a .gitignore file in the top level directory of my project and put this in it:

*.egg
*.pyc
*~

Now I go back to my git-status window and press g (git-status-refresh), and et voila, the clutter disappears.

Committing changes

However, I notice that the hello/main.py is marked as ‘Added’ but not Uptodate. What gives? It seems git-commit-file commits only one file. Oops.  What I should have done is what I do now:

  • ‘M’ to mark all files (.gitignore and hello/main.py)
  • c to commit all files. (Type in log message)
  • g to refresh. (Shows ‘No changes’ and no files are listed)
  • t u (git-toggle-show-uptodate; now lists all files, even those which are uptodate)
  • q (git-status-quit)

Diffing changes

I edit main.py and speaker.py. Going to git-status, and pressing ‘=’ over a file gives me the diff against the committed version.

I can also mark multiple files with ‘m’, or all files with ‘M’. Now pressing ‘=’ diffs all those files.

Committing selected changes

Let’s say I’ve added a bunch of code to some files that I want to commit. I’ve also toggled a gloabl DEBUG variable to aid in my own testing, and I don’t want to commit that particular change. In subversion or cvs I’d have to ensure that the flag was set to its proper state and then commit the whole thing together. However in git, I can commit some changes (patches) and ignore others. This cannot be done from within emacs though. I have to switch to the command line and do a ‘git add –interactive’, which gives me a menu system to add selective ‘hunks’.

Back to the command line, I do a git diff –cached to see what I am going to commit. I don’t like it. I want to start choosing the changes I want to commit again:

  • git reset –mixed (reset the index)
  • git add -i (choose what to add, again)
  • git diff –cached (see what I am gonna be committing0
  • git diff (see what else is there in my WC that I have not scheduled to be committed)
  • git commit (type log and commit)

Note:

Working of git-reset and git-diff

Working of git-reset and git-diff

Up and running with Emacs, Erlang, and Distel

Umm… did I mention that I yearn for hacking Erlang code, too? Following Bill Clementson’s lead, I set up  Emacs+Erlang on my Ubuntu hardy box.

Step 1: Install erlang and erlang-mode

I used the version of erlang and erlang-mode for emacs in Ubuntu’s repositories. I might come to regret it some day, but for now the following was just too simple to resist:

$ sudo aptitude install erlang
# The above will install erlang-mode too;
# if it does not just "apt-get install erlang-mode"

Step 2: Configure erlang-mode in Emacs

I just put the following in my .emacs:

;; Erlang-mode
(require 'erlang-start)
(add-to-hook 'erlang-mode-hook
             (lambda ()
            ;; when starting an Erlang shell in Emacs, the node name
            ;; by default should be "emacs"
            (setq inferior-erlang-machine-options '("-sname" "emacs"))
            ;; add Erlang functions to an imenu menu
            (imenu-add-to-menubar "imenu")))

Now I can launch Emacs and open an erlang file, which puts me in Erlang mode.  I can start and Erlang shell with “C-c C-z”, compile the Erlang code with “C-c C-k”, and view the compilation result in the Erlang buffer (if the buffer is hidden) with “C-c C-l”.  Or, I can just switch to the Erlang shell with “C-c C-z”.  The customizations above enable me to get a menu item in the Emacs menubar with a list of functions defined in the file I am visiting, which is a big help. I am used to “ecb-mode” when coding Python; ECB does not grok Erlang yet, so the imenu is a handy substitute when coding Erlang.

Step 3: Install Distel

Distel is to Erlang and erlang-mode what SLIME is to Lisp and lisp-mode.

To get distel:

$ cd ~/.emacs.d/
$ svn co http://distel.googlecode.com/svn/trunk/ distel
$ cd distel
$ make
$ cd doc
$ make postscript && make postscript # must run twice
$ make info && sudo make install # install the Info documentation
$ info distel # read the distel info documentation

Step 4: Configure Emacs to use Distel

I just need to put this in my .emacs:

(push "/home/parijat/.emacs.d/distel/elisp/" load-path)
(require 'distel)
(distel-setup)

Step 5: Configure Erlang

Distel is designed to work in a distributed Erlang system. It can connect to specified Erlang nodes. My strategy is to use the standard erlang-mode command “C-c C-z” to start an Erlang shell, and connect to it using Distel by “C-c C-d n”. The latter asks for a nodename, which is going to be “emacs@beowulf” (because beowulf is the short hostname of my laptop).

But before we start using remote Erlang nodes, we should create on each physical machine that we will use a ~/.erlang.cookie file with a password, for inter-Erlang node authentication:

$ echo "secret" > ~/.erlang.cookie
$ chmod 0400 ~/.erlang.cookie

Step 6: Play with Distel

So, now we can launch an Erlang node, either via Emacs using “C-c C-z”, or on the command line using “erl -sname mynode”. Then, from within Emacs, we can connect Distel to this node using “C-c C-d n” specifiying the nodename on the prompt. Now we can ask Distel to interrogate the Erlang node for various things.

C-c C-d l ; list erlang processes
PID/Name              Initial Call                              Reds     Msgs
init                  otp_ring0:start/2                         3821        0
erl_prim_loader       erlang:apply/2                           87239        0
error_logger          proc_lib:init_p/5                          229        0
application_controlle erlang:apply/2                            2501        0
<0.7.0>               proc_lib:init_p/5                           45        0
<0.8.0>               application_master:start_it/4               91        0
kernel_sup            proc_lib:init_p/5                         1498        0
rex                   proc_lib:init_p/5                          493        0
global_name_server    proc_lib:init_p/5                           69        0
<0.12.0>              erlang:apply/2                              25        0
<0.13.0>              erlang:apply/2                               4        0
<0.14.0>              erlang:apply/2                               3        0
inet_db               proc_lib:init_p/5                          129        0
net_sup               proc_lib:init_p/5                          312        0
erl_epmd              proc_lib:init_p/5                          147        0

									

Up and running with Emacs, sbcl, and SLIME

While I normally program in Python, a part of me wants to quickly hack some lisp code every now and then. Especially during those long minutes when my automated tests are running.

I realized that long neglect had left me without a working lisp setup. Here is how I got it up and running again.

Step 1: Get rid of system lisp and slime

The version of sbcl and SLIME in my Ubuntu system is not recent enough for my liking. So I get rid of them:

$ sudo apt-get -y remove sbcl slime common-lisp-controller
$ sudo apt-get -y autoremove # get rid of packages we don't need anymore
$ sudo rm -rf /var/cache/common-lisp-controller

Step 2: Install common requirements

cd ~/
sudo apt-get update
sudo apt-get -y install emacs22
sudo apt-get -y install cvs
sudo apt-get -y install git-core
sudo apt-get -y install darcs
sudo apt-get -y install subversion
sudo apt-get -y install build-essential
sudo apt-get -y install autoconf
sudo apt-get -y install curl
sudo apt-get -y install sbcl
sudo apt-get -y install texinfo
sudo apt-get -y install tetex-bin
sudo apt-get -y install xloadimage

Step 3: Get clbuild

$ mkdir lisp
$ cd lisp
# for fresh install of clbuild
$ darcs get http://common-lisp.net/project/clbuild/clbuild
$ cd clbuild
# Or, to update existing clbuild
$ cd clbuild
$ darcs pull

Step 4: Get latest SBCL

$ ./clbuild update sbcl
$ cd source/sbcl
$ sh make.sh
$ cd doc/manual
$ make
$ cd ../..
$ echo > ~/.sbclrc
(require :asdf)
(push "/home/parijat/lisp/clbuild/systems/" asdf:*central-registry*)
^D
$

Step 5: Setup SLIME

$ ./clbuild update slime

Now, we need to add this slime configuration to our .emacs file:

(push "/home/parijat/lisp/clbuild/source/slime" load-path)
;; Common Lisp Mode
(setq inferior-lisp-program "/usr/local/bin/sbcl")
(add-to-list 'auto-mode-alist '("\\.lisp$" . lisp-mode))
(add-to-list 'auto-mode-alist '("\\.cl$" . lisp-mode))
(add-to-list 'auto-mode-alist '("\\.asd$" . lisp-mode))
(require 'slime)
(slime-setup)
(eval-after-load "slime"
 '(progn
    (setq slime-complete-symbol*-fancy t
          slime-complete-symbol-function 'slime-fuzzy-complete-symbol
          slime-when-complete-filename-expand t
          slime-truncate-lines nil
          slime-autodoc-use-multiline-p t)
    (slime-setup '(slime-fancy slime-asdf))
    (define-key slime-repl-mode-map (kbd "C-c ;")
      'slime-insert-balanced-comments)
    (define-key slime-repl-mode-map (kbd "C-c M-;")
      'slime-remove-balanced-comments)
    (define-key slime-mode-map (kbd "C-c ;")
      'slime-insert-balanced-comments)
    (define-key slime-mode-map (kbd "C-c M-;")
      'slime-remove-balanced-comments)
    (define-key slime-mode-map (kbd "RET") 'newline-and-indent)
    (define-key slime-mode-map (kbd "C-j") 'newline)))
(add-hook 'lisp-mode-hook (lambda ()
                           (cond ((not (featurep 'slime))
                                  (require 'slime)
                                  (normal-mode)))
                           (indent-tabs-mode nil)
                           (pair-mode t)))

Note: pair-mode is a nice package for inserting balanced parentheses, quotes, braces and square-brackets, etc., which I installed earlier by hand.

Now we can launch emacs, and start slime with “M-x slime”.

Caveat

For some reason, after doing the above, slime would still not start. Doing a “M-x slime” would throw me into the sbcl debugger with the error message that sbcl could not find “/usr/share/common-lisp/systems/slime/swank-loader.lisp”, or something like that (I could be wrong about the exact path). After much head-scratching (and grepping my ~/lisp directory), I figured out I had to change the emacs customization variable slime-backend. This can be done by doing “M-x customize-group RET slime-lisp RET” and changing the value of “Slime Backend” to simply “swank-loader.lisp” (the file should be in the same directory as “slime.el”). We can also give an absolute path to the file. I found that the value of this variable was set wrongly (an artifact of using apt-get installed slime, I guess) and hence sbcl was not able to load the correct file.

Syncing Nokia N95 with a Linux desktop

Finally found a blog that indicates that you can use an N95 like a business pda without being shackled to a Windows machine: http://davehall.com.au/blog/dave/2007/11/18/my-new-toy-nokia-n95

I am giving it a spin. Will see how it works out.

Ubuntu: "bad signature" problems prevent apt-get update from working

When running a routine package update, I sometimes get errors like this:

$ sudo apt-get update
...
W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used.GPG error: http://security.ubuntu.com hardy-security Release: The following signatures were invalid: BADSIG 40976EAF437D05B5 Ubuntu Archive Automatic Signing Key
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/hardy-security/Release
W: Some index files failed to download, they have been ignored, or old ones used instead.
W: You may want to run apt-get update to correct these problems

This ubuntu bugs page suggests this:

$ sudo apt-get update -o Acquire::http::No-Cache=true

but the problem persists.

Solved it! By deleting the incorrectly signed Release and Release.gpg files that were downloaded in the last try:

$ sudo rm -f /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_hardy-security_Release*

Now apt-get is successful.

I got the idea above from the instructions in comment 20 to the bug

Building and installing a local debian package

Recently, I ran into trouble with the ‘python-zeroc-ice’ package on Ubuntu 8.04. It used to work fine on Ubuntu 7.10.

Some digging revealed that the package python-zeroc-ice-3.2.1-2 claims it supports python2.5, but the C module for ICE is built against python2.4.

So I compiled the package and installed it locally. This is how:

1. Downloaded the source package:

$ apt-get source python-zeroc-ice

2. Installed all other packages needed to *build* this pacakge:

$ sudo apt-get build-dep python-zeroc-ice

3. Fixed the problem:

$ cd zeroc-ice-python-3.2.1 # yes, the source package/dir is called zeroc-python-ice
$ vim config/Make.rules # set "PYTHON_VERSION = python2.5" in the appropriate place.
$ diff -u zeroc-ice-python_3.2.1/config/Make.rules zeroc-ice-python_3.2.1-new/config/Make.rules
--- zeroc-ice-python_3.2.1/config/Make.rules    2007-08-09 03:00:54.000000000 +0800
+++ zeroc-ice-python_3.2.1-new/config/Make.rules        2008-08-11 20:13:09.000000000 +0800
@@ -69,7 +69,8 @@
     # version used for building the Ice extension, then set PYTHON_VERSION
     # to "python2.3" or "python2.4".
     #
-    PYTHON_VERSION     ?= $(word 1,$(notdir $(wildcard /usr/include/python2.[345]*)))
+    #PYTHON_VERSION    ?= $(word 1,$(notdir $(wildcard /usr/include/python2.[345]*)))
+    PYTHON_VERSION      = python2.5
     ifeq ($(PYTHON_VERSION),)
       python_darwin_home = /System/Library/Frameworks/Python.framework/Versions/Current
       PYTHON_VERSION   = $(word 1,$(notdir $(wildcard $(python_darwin_home)/include/python2.[345]*)))
$ dch -i # bring up the changelog with new entry
# Added a changelog messsage.  The package version, at the top of the changelog, is now python-zeroc-ice-3.2.1-2ubuntu1
$ dpkg-buildpackage -uc -us

4. Install the new package

$ cd ..
$ dpkg -i python-zeroc-ice_3.2.1-2ubuntu1_i386.deb

And we are done. However, the change above means Ice won’t be available for python2.4 anymore.

Blogging from GNOME Blog

After several futile attempts at blogging from within Emacs to Blogger.com, I found the GNOME Blog applet and am using it now. It is not emacs, but at least it is less clunky than the blogspot web interface.

Linux and Mobile

The Open Handset Alliance, led by Google, is developing and releasing the Android mobile OS.

But Android is not the only open mobile OS in town. Another noteworthy player is the LiMo foundation. Its mission is to produce an open, Linux based software platform for mobile devices. LiMo has been around longer and there are already LiMo devices in the market.

There are other efforts towards putting Linux on mobile devices:

  • Motorola has a Linux-Java based platform called MOTOMAGX. It is not an open polatform, AFAIK. Also, Motorola’s mobile division’s future is uncertain rigt now.
  • Openmoko is trying to create both a software platform and hardware; they release the design to the hardware as well.
  • Ubuntu Mobile
  • .

Nokia recently joined LiMo. Does this mean a Googls vs Nokia fight in the mobile OS arean? Not to mention, Symbian OS and Windows Mobile are the current dominant players and already have application developer mindshare.