Friday, September 5, 2014

MongoDB warning for running on NUMA machine

When running MongoDB on a Linux server (having multiple physical processors as most servers have these days), you will see the below warning in your log file on startup:

2014-09-04T13:13:35.245-0400 [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-09-04T13:13:35.245-0400 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-09-04T13:13:35.245-0400 [initandlisten] **              numactl --interleave=all mongod [other options]

This check for NUMA based system is done on startup since MongoDB2.0 as running MongoDB or for that matter any database applications can cause poor performance due to excessive swapping.

There is no ideal solution to this problem yet, but a good solution would be to switch off zone reclaim and set a memory interleave policy before starting mongod process. Here are the two steps, both of which need to be done to fully disable NUMA:

  1. Switch off zone reclaim:
    echo 0 > /proc/sys/vm/zone_reclaim_mode
  2. Flush Linux buffer caches just before starting mongod:
    sysctl -q -w vm.drop_caches=3
  3. Set interleave policy for mongod process when starting it:
    numactl --interleave=all <$MONGO_HOME>/mongod <options>

This should solve your problem - but if you want to understand what the issue is and what's really going on - then continue to read below!

Older multi-processor systems were designed to have Uniform Memory Access (UMA) - this essentially means is that all memory is common to all processors and accessed via a common bus. Thus the latency and performance of all processors was the same. This is also referred to as Symmetric Multi-Processing (SMP).

New multi-processor systems were designed to have Non-Uniform Memory Access (NUMA) - each processor has a local bank of memory to which it has very fast access & low latency and it also has access to memory banks of other processors though with a poorer performance & higher latency. This is sometimes also referred to as Cache-Coherent NUMA (ccNUMA).

Linux is aware of its running on a NUMA system and does the following:

  • Probes the system to figure out the physical layout
  • Attaches the memory module to its' corresponding processor to create a node
  • Creates a map (table/matrix) of the cost or weight of inter-node communication between all the nodes
  • Allocates a preferred node for each thread to run on and allocates memory to it, preferably from the same node or from another node if available. If memory is not available on the same node, it is allocated from the available node with the lowest cost
  • Uses zone reclaim to reclaim memory when a node runs out of memory

Zone Reclaim allows Linux SysAdmins to set a more or less aggressive approach to reclaim memory when a zone runs out of memory. If it is set to zero, which is the default, then no zone reclaim occurs. Allocations will be satisfied from other zones / nodes in the system.

Now that you have a simplistic view of what NUMA is, let's understand what our solution actually did.

  1. We switched off zone reclaim - this would reduce paging as allocation of memory is now satisfied from other nodes if required
  2. Flushed Linux's buffer caches - This helps to ensure allocation fairness, even if the daemon is restarted while significant amounts of data are in the operating system buffer cache.
  3. We set memory allocation to interleave for mongod - this spreads allocation of memory evenly on a round-robin basis across all nodes thus spreading out memory usage and reducing paging

For a more in-depth analysis and further reading, check out Jeremy Cole's blog posts:

Wednesday, April 16, 2014

Maven Compile error

Maven Compile Plugin by default uses version 2.0.2 and JDK1.3 as the target for the compilation of Java code if you do not specify the compiler plugin version and the Java source & target versions. Due to this, you may get the below error when compiling:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project scratch: Compilation failure
[ERROR] /home/projects/java/scratch/src/main/java/com/test/[50,26] error: for-each loops are not supported in -source 1.3

To resolve this problem, you need to specify the compiler plugin version and the Java -source and -target versions in the pom.xml of your project as below:


You can figure out the latest version of the Compiler Plugin by reading the version number on the top right of the Compiler Plugin page.

Saturday, April 12, 2014

Throttle task submission in Java - Simple solution

JDK provides a convenient java.util.concurrent.Executors factory with useful methods to return various java.util.concurrent.Executor implementation instance pre-configured with commonly used settings. However, there is no implementation or configuration available for an implementation that throttles task submission. Here I provide my take on a very simple solution for this requirement.

private void submitTasks(List tasksList) {
  final ThreadPoolExecutor executor = new ThreadPoolExecutor(MIN_POOL_SIZE,
                                                             new LinkedBlockingQueue());
  for (Runnable task : tasksList) {
    while (executor.getQueue().size() > MAX_Q_SIZE) {
      // Throttle for WAIT_FOR_SECONDS seconds if queue is full
      try {"Waiting " + WAIT_FOR_SECONDS + " seconds as Queue size is: " + executor.getQueue().size());
        Thread.sleep(WAIT_FOR_SECONDS * 1000);
      } catch (InterruptedException e) {
        // Ignore
  // inform the executor there are no more tasks

Instead of using the factory to get a pre-configured executor, I create my own ThreadPoolExecutor using a LinkedBlockingQueue. This allows us access to the underlying queue for the throttling feature. Before submitting the task, I check if the current queue size is greater than a certain size and if so, then wait for a configured time repeatedly till the queue is smaller.
This implementation is very useful for the scenarios where you have lots of short running tasks that need to be processed by a fixed small number of threads.

Saturday, March 22, 2014

Use EitherMouse for managing multiple mice

I get bored of using the mouse with my right hand so I switch the mouse over to the left hand and prefer to mirror the buttons - i.e. use the left button as right-click and the right button as left-click! I also prefer to have the cursors to be left-handed to complete the whole experience so I use left-handed mouse cursors. After a lot of search, I found the best possible lefty cursors on the MSDN archive site.

This creates hilarious situations when a colleague comes over to my desk and tries to use the left-handed mouse with the right hand. So I was desperately looking for a program that would allow me to have multiple mice with their own settings so that I could have one left-handed mouse for me on the left of the keyboard. And then setup another mouse with the regular settings on my right hand side.

I was not able to find any program that would allow me to do this with any success - till I stumbled upon a very good program - EitherMouse - and the best part - it is absolutely free! It allows individual settings for each connected mouse. Apart from the detailed settings for each mouse, it also very conveniently mirrors the cursors so that you don't need to install special left-handed cursors. After using the program for a few months, I am totally impressed with the way it handles two mice. My setup is very convenient with a lefty mouse and a regular mouse allowing me to switch mouse hands instantly!

The mystery of the missing WhiteSpace in XML Attribute values

My expectation when testing the component that handles the import & export of xml data for the system, was that the input & output xml data should be the same if the component works properly. However, I noticed minor whitespace differences in attribute values between the two xml files and I couldn't find any code that was trimming attribute values during either the import nor the export of xml. A closer look showed that the spaces in attribute values were being trimmed and multiple occurrences of space replaced by a single space when the attribute values were being read in!

This was very intriguing so digging around in the XML Specs, says that in the canonical form of an XML document, attribute values are normalized by the XML processor. The Attribute Value Normalization section further lists out the exact algorithm to be used by the XML processor where-in all occurrences of whitespace are replaced by a space. Furthermore, if the attribute type is not CDATA, then the XML processor must further process the normalized attribute value by trimming leading and trailing space and by replacing sequences of spaces by a single space. Also, there is a separate section to handle new line characters which states that all line-breaks or occurrences of CR & LF must be replaced by a LF character.

Tuesday, July 23, 2013

Split String into Fixed Length chunks using Java

I needed to split a large string into fixed length chunks of equal size using Java Apart from the regular method of looping and doing a substring for the required length, I was wondering in what other way was it possible to achieve the same. Of course, it's Mr. Regex to the rescue for this task at hand! Here was the regex I used to do this:
String largeString = "This is a very large and totally useless and meaningless string";
int chunkLength = 5;
String[] chunks = largeString.split("(?<=\\G.{" + chunkLength + "})");
System.out.println("Number of chunks: " + chunks.length); // Should print 13
System.out.println("Chunk size: " + chunks[0].size()); // Should print 5 == chunkLength
The regular expression is a Positive Look-Behind looking for any chunkLength characters beginning at the position where the last match ended. So the first time around, it matches the beginning of the string and then after that, it keeps matching every set of chunkLength characters.
This makes me think I should write a more detailed post about regular expressions, especially on Look-Ahead and Look-Behind - stay tuned!

Wednesday, May 22, 2013

Free 15+ GB of online storage

Copy is the new online cloud storage service in town.
Sign up for a free account of Copy and you will get 15 GB of storage. And if you use the below referral link, both of us will get an additional 5 GB of extra storage space for free!
So what are you waiting for? Let's all get some more free storage space!

Tuesday, April 30, 2013

Viewing scrolled out logger lines in Eclipse console

Some programs output a lot of text onto the console in Eclipse, especially in debug mode. To ensure that all console output for such programs is available, here are a few Eclipse settings that will be very helpful. Go to Window > Preferences > Run/Debug > Console and change the following:
  1. Fixed width console: un-check this property
  2. Limit console output: un-check this property
  3. Displayed tab width: set this to 4
Save these changes and run the program again. Now you can see the specific logger lines that you want to monitor even if they scroll by and go out of the console.

Thursday, April 18, 2013

Java 5 - Summary of new & important features

Here is a summary of the useful features that were introduced in Java 5.

Generics simplify code
Adds compile-time type safety and eliminates the necessity for type casting without any performance hit

Enhanced For loops - also known as For/Each
Simplifies looping code but does not make the iterator visible. Loops through each entry of the collection/array returning one properly cast value at a time for processing - without requiring to define an Iterator.

Typesafe Enums
Better than public static final String variables - allows us to create enumerated types with arbitrary methods and fields

Metadata Annotations
Reduces coding issues like overloading instead of overriding with the use of pre-defined annotations. Allows
for a more declarative style of programming reducing boiler-plate code with user-defined annotations.

Varargs provides flexibility
Methods can accept a multiple number of parameters defined at runtime with a few restrictions - they have to be the last set of parameters of the method and all have to be of the same data type. The ellipsis "…" is used to indicate that the argument might appear a variable number of times.

Auto Boxing / Unboxing
Implicit conversion between primitives and their wrapper classes - convenient, but can have lower performance as primitives are stored on the stack while the actual objects are stored in the heap.

Synchronization changes in java.util.concurrent package
Introduction of Locks concept to provide better semantics than "synchronized" keyword and many more useful features like Read/Write locks.

Static import
Makes code more readable and reduces redundant typing for qualifying the static members (methods & fields) of a class with the class name in each occurance

Performance improvements with better GC, StringBuilder, etc

Formatter allows for better printing with printf
No more clumsy println's with string concatenation - use C-style printf with format strings.

Scanner simplifies basic parsing
Easier than String.split and Integer.parseInt

These are too numerous and important to detail all of them in a single post - so each will be covered in detail with code examples in posts of their own soon.

Check out the Summary of Java 7 features that make Java even better!

Wednesday, April 10, 2013

Six Sigma - a brief introduction

Sigma is the 18th letter of the Greek alphabet and among other things, the lower case representation is used to denote the Standard Deviation of a population or Probability Distribution in Statistics. In simple terms, this denotes how much of a deviation there is from the perfect situation.

Enough already about plain Sigma, now let's look at Six Sigma.
Six Sigma is a set of data-driven tools and strategies for improving processes - originally developed by Motorola in 1985. It became more popular after Jack Welch made it the standard process to be used in GE in 1995.

The main philosophy behind Six Sigma is to use a data driven approach to measure the number of defects in the process and then figure out a way to systematically reduce them as close to zero as possible. To achieve Six Sigma quality, the defects should be reduced to less than 3.4 per million opportunities. Say for example, if a million parts are produced by a factory in a week, then it would be running at Six Sigma if less than 3.4 (3 or less) parts turn out to be defective in that week.

It was originally intended for improving existing manufacturing processes but later on enhanced to have a methodology for designing new processes as well.
The two methodologies which are composed of five phases each, are (quoted from Wikipedia):
DMAIC : used for improving an existing process.
DFSS / DMADV : used for creating/designing new product or process.


The DMAIC project methodology consists of the following five phases:
Define the problem and the project goals, specifically.
Measure key aspects of the current process and collect relevant data.
Analyze the data to investigate and verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered. Seek out root cause of the defect under investigation.
Improve or optimize the current process based upon data analysis using techniques such as design of experiments, poka yoke or mistake proofing, and standard work to create a new, future state process. Set up pilot runs to establish process capability.
Control the future state process to ensure that any deviations from target are corrected before they result in defects. Implement control systems such as statistical process control, production boards, visual workplaces, and continuously monitor the process.


The DMADV methodology is also known as DFSS - "Design For Six Sigma" and consists of the following five phases:
Define design goals that are consistent with customer demands and the enterprise strategy.
Measure and identify CTQs (characteristics that are Critical To Quality), product capabilities, production process capability, and risks.
Analyze to develop and design alternatives
Design an improved alternative, best suited per analysis in the previous step
Verify the design, set up pilot runs, implement the production process and hand it over to the process owner(s).

Six Sigma denotes four roles for its successful implementation as shown below:
Executive Leadership or top management who are responsible for creating the Six Sigma vision and to empower the employees to bring about this change.
Champions who take responsibility for Six Sigma implementation across the organization in an integrated & consistent manner and also identifying projects/functions for Six Sigma.
Master Black Belts who act as in-house coaches on Six Sigma & devote all their time only to Six Sigma. They assist Champions and guide Black Belts & Green Belts and work on ensuring consistent application of Six Sigma across various functions and departments.
Black Belts operate under Master Black Belts to apply Six Sigma methodology to specific projects and devote all their time to application & execution of Six Sigma for specific projects.
Green Belts are the employees who take up Six Sigma implementation along with their other job responsibilities and work under the guidance of Black Belts.


Related Posts Plugin for WordPress, Blogger...