1-year-old thinks a magazine is a broken iPad

MySQL and Index Woes

I have often noticed that altering indexes on large tables in MySQL is a very time consuming process. I’d chalked it up to “just how things are” and account for that when doing database migrations. Possibly because I’d usually *add* an index on the table not drop it.

But apparently there there is a design flaw in MySQL 5.0 and before with respect to how it handles indexes that caused dropping indexes on tables very slow, and also adding a new index on the table slower than it needed to be. Gory details here: http://bugs.mysql.com/bug.php?id=2364

But in 5.1, at least dropping table ought to be near instantaneous. Adding an index? Umm… it seemed the old behaviour was:

Given the same MyISAM table T having four indexes (ndx1,ndx2,ndx3,ndx4)
and you want to ‘alter table T add index ndx5 (…);’ here is exactly
what happens (from: http://lists.mysql.com/mysql/202489)

1) MySQL copies T.MYD to a temp table, i.e., S.MYD and a zero byte S.MYI.
2) MySQL does ‘alter table S add index ndx1 (…);
3) MySQL does ‘alter table S add index ndx2 (…);
4) MySQL does ‘alter table S add index ndx3 (…);
5) MySQL does ‘alter table S add index ndx4 (…);
6) MySQL does ‘alter table S add index ndx5 (…);
7) MySQL deletes T.MYD and deletes T.MYI
8) MySQL renames S.MYD to T.MYD, and renames S.MYI to T.MYI

Note that each of the “alter table” step involves copying the table to a temp table with no indexes, applying all existing indexes one by one, and then applying the new index…. so in the above case:

In fact, for table T with no indexes and you want to add N indexes
MySQL will copy the MYD N times
MySQL will copy the MYI N times
MySQL will run ‘alter table add index’ N(N+1)/2 times if adding an index
MySQL will run ‘alter table drop index’ N(N-1)/2 times if dropping an index

So here is a better way to add indexes to a table:

1) create table T1 like T;
This creates an empty table T1 with indexes ndx1,ndx2,ndx3 and ndx4.
2) alter table T1 drop index ndx3;
This drops index ndx3 on the empty T1, which should be instantaneous.
3) insert into T1 select * from T;
This will populate table T and load all three(3) indexes for T1 in one pass.
4) drop table table T;
5) alter table T1 rename to T;

Starhub MaxOnline and speedtest.net

I have a Starhub MaxOnline Premium connection. Ostensible it provides a download speed of upto 30Mbps. I am happy to pay the extra price for the increased bandwidth. But am I getting more bandwidth for muy money? How much more?

Spring Java Config and ProxyFactoryBean

I really like spring remoting. I really like spring java config. Recently I had to marry the two. Here is how.

I have a value-object:

public class Datum
        implements Serializable
    {
    private int id;
    private String name;

    public Datum(int id, String name)
        {
        this.id = id;
        this.name = name;
        }

    public int getId()
        {
        return id;
        }

    public String getName()
        {
        return name;
        }

    @Override
    public String toString()
        {
        return "Datum{" + "id=" + id + "name=" + name + '}';
        }

    }

I have a service that produces Datum objects. The service interface is:

public interface IDatumProducer
    {
    public Datum produce();
    }

and a sample implementation might be:

public class RandomDatumProducerImpl
        implements IDatumProducer
    {
    private static final Logger log =
            LoggerFactory.getLogger(RandomDatumProducerImpl.class);

    private Random rnd = new Random(System.currentTimeMillis());

    @Override
    public Datum produce()
        {
        int i = rnd.nextInt();
        Datum datum = new Datum(i, String.valueOf(i));
        log.info("Produced {}", datum);
        return datum;
        }

    }

Here is a client for the service:

public class DatumProducerClient
    {
    private static final Logger log =
            LoggerFactory.getLogger(DatumProducerClient.class);

    private final IDatumProducer producer;

    public DatumProducerClient(IDatumProducer producer)
        {
        this.producer = producer;
        }

    @PostConstruct
    public void init()
        {
        while (true)
            {
            try
                {
                Datum datum = producer.produce();
                log.info("Got {}", datum);
                Thread.sleep(1000L);
                }
            catch (InterruptedException ex)
                {
                log.error("Interrupted", ex);
                }
            }
        }
    }

Now I want to run the DatumProducerClient and the IDatumProducer on different machines. The client should connect remotely to the producer, invoke the produce method, and do its stuff.

We can expose the service easily using Spring. The spring remoting (JMS) manual section gives the spring configuration XML to do this, but no example for doing this in Java Config. Turns out there is one small detail to take care of when using Java config.

Producer (Server)

Here is the application config for the producer side:

@Configuration
public class ProducerConfig
    {

    @Bean
    public AbstractMessageListenerContainer messageListenerContainer()
        {
        SimpleMessageListenerContainer container =
                new SimpleMessageListenerContainer();
        container.setConnectionFactory(jmsConnectionFactory());
        container.setDestination(jmsQueue());
        container.setMessageListener(jmsInvokerServiceExporter());
        container.setConcurrentConsumers(3);
        return container;
        }

    @Bean
    public JmsInvokerServiceExporter jmsInvokerServiceExporter()
        {
        JmsInvokerServiceExporter serviceExporter =
                new JmsInvokerServiceExporter();
        serviceExporter.setService(datumProducer());
        serviceExporter.setServiceInterface(IDatumProducer.class);
        return serviceExporter;
        }

    @Bean
    public ConnectionFactory jmsConnectionFactory()
        {
        ActiveMQConnectionFactory baseConnectionFactory =
                new ActiveMQConnectionFactory();
        baseConnectionFactory.setBrokerURL("tcp://localhost:61616");
        return baseConnectionFactory;
        }

    @Bean
    public Queue jmsQueue()
        {
        ActiveMQQueue queue = new ActiveMQQueue("dataChannel");
        return queue;
        }

    @Bean
    public IDatumProducer datumProducer()
        {
        return new RandomDatumProducerImpl();
        }

    }

And here is a main method that kicks off our application.

public class PublisherMain
    {
    public static void main(String[] args)
        {
        AnnotationConfigApplicationContext ctx =
                new AnnotationConfigApplicationContext(ProducerConfig.class);
        }
    }

Consumer (Client)

Here is how the consumer is wired up.

@Configuration
public class ConsumerConfig
    {

    @Bean
    public DatumProducerClient datumProducerClient()
        {
        return new DatumProducerClient(datumProducer());
        }

    @Bean
    public IDatumProducer datumProducer()
        {
        // producerFactory.getObject() will return null if we don't define
        // it as a separate bean; if we copy over the code from
        // jmsInvokerProxyFactoryBean here, it won't work: the proxy won't
        // get "enhanced" by spring, and its getObject() method will always
        // return null.
        JmsInvokerProxyFactoryBean producerProxy = jmsInvokerProxyFactoryBean();
        return (IDatumProducer) producerProxy.getObject();
        }

    // We can't fold this code into datumProducer above
    // The proxy bean must be defined separately, otherwise spring won't get
    // a chance to enhance it.
    @Bean
    public JmsInvokerProxyFactoryBean jmsInvokerProxyFactoryBean()
        {
        JmsInvokerProxyFactoryBean producerProxy = new JmsInvokerProxyFactoryBean();
        producerProxy.setServiceInterface(IDatumProducer.class);
        producerProxy.setConnectionFactory(jmsConnectionFactory());
        producerProxy.setQueue(jmsQueue());
        return producerProxy;
        }

    @Bean
    public ConnectionFactory jmsConnectionFactory()
        {
        ActiveMQConnectionFactory baseConnectionFactory =
            new ActiveMQConnectionFactory();
        baseConnectionFactory.setBrokerURL("tcp://localhost:61616");
        return baseConnectionFactory;
        }

    @Bean
    public Queue jmsQueue()
        {
        ActiveMQQueue queue = new ActiveMQQueue("dataChannel");
        return queue;
        }
    }

and here is how the main method on the client side looks like:

public class ConsumerMain
    {
    public static void main(String[] args)
        {
        AnnotationConfigApplicationContext ctx =
                new AnnotationConfigApplicationContext(ConsumerConfig.class);
        }
    }

The important bit to note here is that the JmsInvokerProxyFactoryBean is configured as a bean in its own right. This allows spring to “enhance” it at runtime to do all the proxying for us. If this were not a separate bean, things would not work. For e.g., the following code won’t work:

// Wrong way to use a ProxyFactoryBean
    @Bean
    public IDatumProducer datumProducer()
        {
        JmsInvokerProxyFactoryBean producerProxy = new JmsInvokerProxyFactoryBean();
        producerProxy.setServiceInterface(IDatumProducer.class);
        producerProxy.setConnectionFactory(jmsConnectionFactory());
        producerProxy.setQueue(jmsQueue());
        return (IDatumProducer) producerProxy.getObject(); // Wrong: getObject always returns null
        }

That’s all!

Buttons that don’t look like buttons: a rant on usability

As part of work, I recently was installing and configuring an open source
software. I was in a big hurry to get something done. This software was
a helper program; it did something awfully nice and useful and which
would make “disciplined” programmers jump with joy. But I was not
jumping with joy. I had something to get done. I needed this software,
but I was not prepared to delve deeply into it. I did not want to learn
too much about it. I just needed it to work, bare minimum, no fancy
stuff, so that I could get on with writing my code.

I suppose I was mentally prepared to fight the software. If it actually
was performing such a nice and useful function, it ought to be
complicated and self-important. “Read my README”, I imagined it
saying. “Then read the INSTALL.txt, and Getting Started, and then
Adminstrator’s Guide…”. I was hoping to wing it. I’d try to start
it, see what broke, try to configure that, see what else broke, try to
fix that, and so on until I won and the software lost the game of
Yet-Another-Installation-Nightmare.

My fears turned out to be grounless.
Installing it was a breeze. One just unpacked the
software and ran a start script. All dependencies were already included
so I did not have to hunt down any. It just started up, presenting a
web-based management UI. It was already configured with sensible
defaults to get a new user up to speed quickly. Wow!

My next step was to create a user account. I brought up the web
management console in firefox. I clicked on the “Users” link, and was
presented with a list of existing user accounts. I could not see how
to create a new user. I could see how to attach users to roles, I
could see how to search for users using their name, role, email etc.
But for the life of me I could not figure out how to create a new
user account.

When I was just about to admit defeat and turn to the “User Manual”, the
answer jumped out of the page and smacked me in the forehead. There it
was, clear as day, a button labeled “Create New User”. I had been
staring at various links and trying everything on that page, expecting
that this step would be as easy as the install part was, and getting
slightly annoyed that it wasn’t. I had been clicking at everything but
this button. In fact, I had given this button a good long stare and
looked around it to find a link that I could click.

Had my brain suddenly turned into putty? What is so hard about clicking
a dar***d button?

There is nothing hard about clicking a button. Except that the button
in question did not look like a button. The button looked like a
flat, light gray box. When I hovered the mouse over the box, the cursor
did not change into something that indicated “click me”. Basically,
someone had gone out of their way to create a CSS style that made
the button obscure.

Why?

Because it looked “better”? Because it was there to be styled? Just
because? Beats me. The point is that this beautifully useful and easy
to setup software had a GUI blooper right there in the management
console on one of the most commonly used functions. In other words,
this very useful software, which was insanely great in so many other
ways, had a usability issue.

Same day. Another web page. This was a commercial web site that served
up search results arranged alphabetically. There was a nice, helpful
row of navigation links at the top, labeled “A”, “B”, and so one all the
way to “Z”, to allow the user to see results starting with that lette.
The letters were surrounded in a nice, colored box that became highlighted
when your mouse hovered over the box.

So I moused over to the box containing “P” and when it was highlighted,
I clicked.

Nothing happened.

So I double-checked and found that while the box gets highlighted when
your mouse hovered over it, only the letter itself was clickable, not
the box. The “target area” presented by the letter is significantly
smalled than the box, forcing you to be precise about where you click.
There might as well not have been any box. In fact, the box was
downright misleading. It looked good. But it lulled my busy brain into
trying to click something that was not clickable. Another usability
issue.

Usability has been on my mind recently. (Gee, can you tell?). It is
amazing that there is so much software and so many websites out there
with small, niggly usability issues that are sucking away seconds,
minutes or hours of their users life as they grumpily are trying to get
their jobs done. It is sad, really. The code is there. It works. But
the path to executing that code is strewn with brambles and thickets of
usability issues.

Programmers need to pay attention to the usability of their UIs. Most
books, tutorials and manuals focus on making the UI look good, but few
point out how to make the UI more usable. Most people don’t read the UI
design guidelines of the platform they are working on. And there are
probably no standard guidelines for designing for the web. Web
technologies are changing too fast anyway for any technology focused
guidelines to remain relevant for long. The core principles are taught
in the discipline of Interaction Design. But I don’t think many people
know what Interaction Designers do. I myself didn’t until very
recently.

There really ought to be more interaction designers out there and they
ought to have more work. Programmers and managers should be paying
attention to the usability of their software and its interface.

But for now, I am too busy slapping together functionality to pay
due attention to how it will end up getting used. Oh, I think about
this issue and try to do something about it, for sure. I try to design
the software, imagining the end user and his/her goals in mind.

But that is not enough. User interviews, mock-ups, usability testing
are some of the tools that are involved in producing usable software. I
did not budget for it; I did not set up any time for it. And I wonder
if anyone would understand me if I told them I wanted to take a flight
to meet our users (internal business folks) and do a usability testing.

Or maybe they would. One of these days, I am going to try.

CPU consumption by idle JVM and how to reduce it

http://blogs.sun.com/nickstephen/entry/java_why_does_my_jvm

SaaS step-by-step

A short and sweet article on JavaWorld (RoadMap to SaaS) discusses the steps involved in taking a run of the mill web-app and making it a distributed service oriented system. Not heavy on the details but IMHO presents the things an architect needs to be thinking about in a very compelling progression. So compelling, in fact, that I must have another look at it when I am more awake.

How to choose when to start a project

I have observed this happen several times: a development team or teams are busy working on several projects and are fully occupied. Suddenly management thinks of an attractive new project and want it to be done quickly.

What happens next?

SEMAT, Software Engineering, and the fashion industry

Apparently there is going to be a revolution in the software development world, backed by some very illustrious names in the industry. In the words of Ivar Jacobson, one of the people behind it:

We are some people who have observed software engineering theory and practice of the past decades and have realized that it is now time to revitalize this discipline. We have been quietly planning a “revolution”.

And here is the “Call for Action” statement for this revolution (called SEMAT — Software Engineering Method and Theory):

Software engineering is gravely hampered today by immature practices. Specific problems include:

• The prevalence of fads more typical of fashion industry than of an engineering discipline.
• The lack of a sound, widely accepted theoretical basis.
• The huge number of methods and method variants, with differences little understood and artificially magnified.
• The lack of credible experimental evaluation and validation.
• The split between industry practice and academic research.

We support a process to refound software engineering based on a solid theory, proven principles and best practices that:

• Include a kernel of widely-agreed elements, extensible for specific uses
• Addresses both technology and people issues
• Are supported by industry, academia, researchers and users
• Support extension in the face of changing requirements and technology

The list of signatories to the “revolution” is quite impressive. See for your self on their web page. Sounds good? But not everyone agrees, as you can see on this blog. The author of the blog has an issue with the very framing of the problem:

But this is an entirely misguided movement. Let’s analyze the call for action, bit by bit. It begins:

Software engineering is gravely hampered today by immature practices. [...]

[...]

Never mind that people in the community (most prominently Tom DeMarco) have spoken against this paradigm, or that software development has little to do with most established engineering disciplines: the people behind SEMAT have chosen to ignore these criticisms and move on.

Fair enough. This critique reminds me of this really old paper, called “Masterpiece Engineering” that you might have heard of. Apparently, there were people doubting whether software development is really engineering even when the term was first coined circa 1968.

But what interests me is this statement by SEMAT:

The prevalence of fads more typical of fashion industry than of an engineering discipline.

So if Software Engineering is not engineering, could it be fashion? Let’s see. Here are my observations:

• Formal study: (a) Electrical Engineer: formal degree required; (b) Software Engineer: formal degree not required; (c) Fashion Designer: formal degree not required
• Work: Engineers: work hard; (b) Software developers: work harder; (c) Fashion Designers: while they are apprentices, work their ass off; as soon as they “become” a designer, stop calling it work and start calling it art; the amount of work/art that needs to be done everyday is inversely proportional to how good you are considered
• Respect: (a) Electrical Engineer: gets some respect from the people who work for him, for his knowledge; gets respect from people he works for (the “business folks”) only if he can keep the cost low; (b) Software Engineer: gets no respect from anyone, including the janitor services, unless he accidentally makes some money from a dot.com IPO; (c) Fashion Designer: gets respected both by people who work for him, and people who he works for, directly proportional to how much he/she is exploiting them
• Fame: (a) Electrical Engineer: not famous, with rare exceptions of people who have managed to make something that has the astonishing capability of being really appealing to customers and their company’s marketing department at the same time; (b) Software Engineer: not famous as long as they are doing the right thing; you need to write a worm that brings down the ‘net to be famous and maybe not even then (c) Fashion Designer: often famous, especially if the product simultaneously manages to be feckless (you don’t know whether its a skirt or a bandanna) and useless (does not cover the body)
• Discipline: (a) Electrical Engineer: practices great discipline in work; (b) Software Engineer: is expected to work with discipline during office hours and continue to work with passion until midnight; (c) Fashion Designer: discipline is death!
• Creativity: (a) Electrical Engineer: as his experience grows, tries his hand at more creative things; (b) Software Engineer: to start with, because he does not know much yet, everyone around is too busy to teach, there is no training budget, and anyway the product has to ship by month end with no time to learn anything, has to be creative by using the only tool he know to do everything; only after gaining much experience can he stop being creative and actually do thing properly; (c) Fashion Designer: your creativity is defined to be inversely proportional to how many people can afford your clothes

So, from the above, it is clear, fad or no fad, a Software Engineer’s life is nothing like a Fashion Designer’s. It is also not like an Electrical Engineer’s.

P.S.: while I have a Bachelors and Master degree in Electrical Engineering, and I have been developing software professionally for some time now, my profound knowledge of the fashion industry is gained entirely from the movie The Devil Wears Prada.

Monitoring, knowledge and teamwork

Jeremy Zawodny, MySQL guru, tells a story about:

One of the frustrating things about building and running backend infrastructure services is that things break sometimes — often in unexpected ways. And when they do, you can spend untold hours chasing down various ideas, most of which are dead ends, before eventually solving the problem.

He starts with a curious MySQL login failure and ends up with IRQs and boot time kernel options.

Scary.

Read the story here: part I, part 2.