Saturday, March 24, 2018

Copy specific files from within nested folders

I had a situation where I needed to save all log files that were present in nested sub-folders and copy them over to another folder for analysis.
There are many ways to do this - but here is the simplest that I could figure out using rsync:
rsync -aihv --include='*/' --include='*.log* --exclude='*' /source_folder/to/search/in/  /destination_folder/to/copy/to/

Important thing to note here is the order of include and exclude flags passed to rsync command - it is sensitive to the order in which these flags are provided.
First include flag '*/' makes rsync look into subfolders and the second flag selects all files of the format '*.log*'.
The next exclude flag excludes any file not selected as yet and then copies them over to the destination folder.
You can provide more include flags before the exclude flag to select multiple types of files as per your requirement.

Rename multiple files in one command

If you want to rename multiple files, the trusty old Unix "mv" command can't help as it can rename only one file at a time.
This requires us to bring out something different from our Unix arsenal - the "rename" command!
Use the rename command in conjunction with find to rename multiple files:
find /source_folder/to/search/in/ -name 'myapp*.log.201*' -exec rename -v .log. .oldlog. {} \; >> /temp/rename.log

Here we are searching for files named 'myapp*.log.201*' and renaming them to 'myapp*.oldlog.201*'
In case you need to do this on multiple servers, use the for loops trick!

Running same set of commands on multiple servers using for loop

Developers as usual are very lazy and so instead of going to each server to grep for an error string in the log file and creating a report, here is a simple way to collect all the necessary log lines.
Create a file containing a list of server names on which we want to check the logs - one on each line and then run this for loop if using bash:

for s1 in $(cat servers_list.txt); do
  ssh -q ${s1} "grep 'search_string' /search/folder/*/*.log* | sed 's/^/${s1} => /'" >> /tmp/search.log
  # Other commands here if required

If you are using csh, then the for loop syntax is slightly different:

foreach s1 in (cat servers_list.txt)
  ssh -q ${s1} "grep 'search_string' /search/folder/*/*.log* | sed 's/^/${s1} => /'" >> /tmp/search.log
  # Other commands here if required

The code is simple - for each server, ssh to that server and run the grep command to find out the log lines having the specific string. The bigger trick here is to use sed to pre-pend server name to the grepped log line to make the report useful. This should return a file with lines in the format:

server_name => [log line grepped from the log file]


Related Posts Plugin for WordPress, Blogger...