Friday, November 7, 2014

MongoDB keyfile permissions are too open error

Creating an authenticated replicated set requires setting up of a secret key file on all the nodes.
In spite of creating the key file and putting the same file on all the nodes, you may encounter error stating that the permissions are too open:

permissions on /var/mongodb/conf/mongodb-keyfile are too open

 To solve this, here are the two things that you need to do to solve it:

  1.  Check that the owner of the key file and the user running mongod are same.
     Use chown command to change ownership to match if required.
      chown
  2.  Check permissions of the key file - should be 600 or 700.
     Use chmod command to make permissions more restrictive.
     chmod 600

Amazon Echo - your own personal digital concierge!

Amazon has launched nifty device named Echo - which is like a stand-alone "Siri" - but more like a personal digital concierge for the family. The cost is pretty reasonable at $199 and Amazon is running a promotion right now where Prime members can get it for only $99.

Check out the Echo on Amazon - looks pretty interesting but is available by invite only.

Friday, September 5, 2014

MongoDB warning for running on NUMA machine

When running MongoDB on a Linux server (having multiple physical processors as most servers have these days), you will see the below warning in your log file on startup:

2014-09-04T13:13:35.245-0400 [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-09-04T13:13:35.245-0400 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-09-04T13:13:35.245-0400 [initandlisten] **              numactl --interleave=all mongod [other options]

This check for NUMA based system is done on startup since MongoDB2.0 as running MongoDB or for that matter any database applications can cause poor performance due to excessive swapping.

There is no ideal solution to this problem yet, but a good solution would be to switch off zone reclaim and set a memory interleave policy before starting mongod process. Here are the two steps, both of which need to be done to fully disable NUMA:

  1. Switch off zone reclaim:
    echo 0 > /proc/sys/vm/zone_reclaim_mode
  2. Flush Linux buffer caches just before starting mongod:
    sysctl -q -w vm.drop_caches=3
  3. Set interleave policy for mongod process when starting it:
    numactl --interleave=all <$MONGO_HOME>/mongod <options>

This should solve your problem - but if you want to understand what the issue is and what's really going on - then continue to read below!

Older multi-processor systems were designed to have Uniform Memory Access (UMA) - this essentially means is that all memory is common to all processors and accessed via a common bus. Thus the latency and performance of all processors was the same. This is also referred to as Symmetric Multi-Processing (SMP).

New multi-processor systems were designed to have Non-Uniform Memory Access (NUMA) - each processor has a local bank of memory to which it has very fast access & low latency and it also has access to memory banks of other processors though with a poorer performance & higher latency. This is sometimes also referred to as Cache-Coherent NUMA (ccNUMA).

Linux is aware of its running on a NUMA system and does the following:

  • Probes the system to figure out the physical layout
  • Attaches the memory module to its' corresponding processor to create a node
  • Creates a map (table/matrix) of the cost or weight of inter-node communication between all the nodes
  • Allocates a preferred node for each thread to run on and allocates memory to it, preferably from the same node or from another node if available. If memory is not available on the same node, it is allocated from the available node with the lowest cost
  • Uses zone reclaim to reclaim memory when a node runs out of memory

Zone Reclaim allows Linux SysAdmins to set a more or less aggressive approach to reclaim memory when a zone runs out of memory. If it is set to zero, which is the default, then no zone reclaim occurs. Allocations will be satisfied from other zones / nodes in the system.

Now that you have a simplistic view of what NUMA is, let's understand what our solution actually did.

  1. We switched off zone reclaim - this would reduce paging as allocation of memory is now satisfied from other nodes if required
  2. Flushed Linux's buffer caches - This helps to ensure allocation fairness, even if the daemon is restarted while significant amounts of data are in the operating system buffer cache.
  3. We set memory allocation to interleave for mongod - this spreads allocation of memory evenly on a round-robin basis across all nodes thus spreading out memory usage and reducing paging

For a more in-depth analysis and further reading, check out Jeremy Cole's blog posts:

Wednesday, April 16, 2014

Maven Compile error

Maven Compile Plugin by default uses version 2.0.2 and JDK1.3 as the target for the compilation of Java code if you do not specify the compiler plugin version and the Java source & target versions. Due to this, you may get the below error when compiling:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project scratch: Compilation failure
[ERROR] /home/projects/java/scratch/src/main/java/com/test/MavenCompileTest.java:[50,26] error: for-each loops are not supported in -source 1.3

To resolve this problem, you need to specify the compiler plugin version and the Java -source and -target versions in the pom.xml of your project as below:


  
    
      org.apache.maven.plugins
      maven-compiler-plugin
      3.1
      
        1.7
        1.7
      
    
  


You can figure out the latest version of the Compiler Plugin by reading the version number on the top right of the Compiler Plugin page.

Saturday, April 12, 2014

Throttle task submission in Java - Simple solution

JDK provides a convenient java.util.concurrent.Executors factory with useful methods to return various java.util.concurrent.Executor implementation instance pre-configured with commonly used settings. However, there is no implementation or configuration available for an implementation that throttles task submission. Here I provide my take on a very simple solution for this requirement.

private void submitTasks(List tasksList) {
  final ThreadPoolExecutor executor = new ThreadPoolExecutor(MIN_POOL_SIZE,
                                                             MAX_POOL_SIZE,
                                                             60L,
                                                             TimeUnit.SECONDS,
                                                             new LinkedBlockingQueue());
  for (Runnable task : tasksList) {
    while (executor.getQueue().size() > MAX_Q_SIZE) {
      // Throttle for WAIT_FOR_SECONDS seconds if queue is full
      try {
        log.info("Waiting " + WAIT_FOR_SECONDS + " seconds as Queue size is: " + executor.getQueue().size());
        Thread.sleep(WAIT_FOR_SECONDS * 1000);
      } catch (InterruptedException e) {
        // Ignore
      }
    }
    executor.execute(task);
  }
  // inform the executor there are no more tasks
  executor.shutdown();
}

Instead of using the factory to get a pre-configured executor, I create my own ThreadPoolExecutor using a LinkedBlockingQueue. This allows us access to the underlying queue for the throttling feature. Before submitting the task, I check if the current queue size is greater than a certain size and if so, then wait for a configured time repeatedly till the queue is smaller.
This implementation is very useful for the scenarios where you have lots of short running tasks that need to be processed by a fixed small number of threads.

Saturday, March 22, 2014

Use EitherMouse for managing multiple mice

I get bored of using the mouse with my right hand so I switch the mouse over to the left hand and prefer to mirror the buttons - i.e. use the left button as right-click and the right button as left-click! I also prefer to have the cursors to be left-handed to complete the whole experience so I use left-handed mouse cursors. After a lot of search, I found the best possible lefty cursors on the MSDN archive site.

This creates hilarious situations when a colleague comes over to my desk and tries to use the left-handed mouse with the right hand. So I was desperately looking for a program that would allow me to have multiple mice with their own settings so that I could have one left-handed mouse for me on the left of the keyboard. And then setup another mouse with the regular settings on my right hand side.

I was not able to find any program that would allow me to do this with any success - till I stumbled upon a very good program - EitherMouse - and the best part - it is absolutely free! It allows individual settings for each connected mouse. Apart from the detailed settings for each mouse, it also very conveniently mirrors the cursors so that you don't need to install special left-handed cursors. After using the program for a few months, I am totally impressed with the way it handles two mice. My setup is very convenient with a lefty mouse and a regular mouse allowing me to switch mouse hands instantly!

The mystery of the missing WhiteSpace in XML Attribute values

My expectation when testing the component that handles the import & export of xml data for the system, was that the input & output xml data should be the same if the component works properly. However, I noticed minor whitespace differences in attribute values between the two xml files and I couldn't find any code that was trimming attribute values during either the import nor the export of xml. A closer look showed that the spaces in attribute values were being trimmed and multiple occurrences of space replaced by a single space when the attribute values were being read in!

This was very intriguing so digging around in the XML Specs, says that in the canonical form of an XML document, attribute values are normalized by the XML processor. The Attribute Value Normalization section further lists out the exact algorithm to be used by the XML processor where-in all occurrences of whitespace are replaced by a space. Furthermore, if the attribute type is not CDATA, then the XML processor must further process the normalized attribute value by trimming leading and trailing space and by replacing sequences of spaces by a single space. Also, there is a separate section to handle new line characters which states that all line-breaks or occurrences of CR & LF must be replaced by a LF character.

Tuesday, July 23, 2013

Split String into Fixed Length chunks using Java

I needed to split a large string into fixed length chunks of equal size using Java Apart from the regular method of looping and doing a substring for the required length, I was wondering in what other way was it possible to achieve the same. Of course, it's Mr. Regex to the rescue for this task at hand! Here was the regex I used to do this:
String largeString = "This is a very large and totally useless and meaningless string";
int chunkLength = 5;
String[] chunks = largeString.split("(?<=\\G.{" + chunkLength + "})");
System.out.println("Number of chunks: " + chunks.length); // Should print 13
System.out.println("Chunk size: " + chunks[0].size()); // Should print 5 == chunkLength
The regular expression is a Positive Look-Behind looking for any chunkLength characters beginning at the position where the last match ended. So the first time around, it matches the beginning of the string and then after that, it keeps matching every set of chunkLength characters.
This makes me think I should write a more detailed post about regular expressions, especially on Look-Ahead and Look-Behind - stay tuned!

Wednesday, May 22, 2013

Free 15+ GB of online storage


Copy is the new online cloud storage service in town.
Sign up for a free account of Copy and you will get 15 GB of storage. And if you use the below referral link, both of us will get an additional 5 GB of extra storage space for free!
So what are you waiting for? Let's all get some more free storage space!

https://copy.com?r=yLbtQq

Tuesday, April 30, 2013

Viewing scrolled out logger lines in Eclipse console

Some programs output a lot of text onto the console in Eclipse, especially in debug mode. To ensure that all console output for such programs is available, here are a few Eclipse settings that will be very helpful. Go to Window > Preferences > Run/Debug > Console and change the following:
  1. Fixed width console: un-check this property
  2. Limit console output: un-check this property
  3. Displayed tab width: set this to 4
Save these changes and run the program again. Now you can see the specific logger lines that you want to monitor even if they scroll by and go out of the console.

LinkWithin

Related Posts Plugin for WordPress, Blogger...