Category: Snippets

Create a select menu from subfolders

If you want to run some code on a directory, but maybe not always the same directory, then generating a menu can be very handy.
# The menu we are making looks like this
[me@parentfolder]$ directoryPicker $pwd
1) Subfolder_1
2) Subfolder_2
3) Subfolder_3
#? 
To make a generic menu, call select with an array, and it displays the choices. You can get complicated and have different return or output values for each of the choices but for this example, we are just setting a variable, DIRECTORY, which is returned to the calling function.
# The select function portion of the example
select dir in "${dirs[@]}"; do
  # This will be run only on the user's choice, the function itself 
  # handles displaying the array values as a menu
  DIRECTORY="${PARENT}/${dir}"
  break;
done
You can declare an array with values for each item under a directory by using a subshell to call ls.
# make an array using a directory listing
declare -a dirs=($(ls -d *))

# Want to check what is in here?  Echo all the entries with [*]
echo ${dirs[*]}
But you probably want to do a little filtering to remove files, since passing them to your calling function won’t do much and it makes the menu less readable.

The basics of looping through a bash array probably look familiar, but for the iteration note that ${array[@]} will give you each array value in its own variable, and ${array[*]} will return them all as a single variable.
# a basic array loop
for dir in "${dirs[@]}"; do
  echo ${dir}
done
To remove an item from an array, you want to unset that index, so you’re going to give this loop an index and increment the index using let.
index=0
for dir in "${dirs[@]}"; do
  if [ ! -d "${PARENT}/${dir}" ]; then
    # unset an array element by referencing the variable _name_, 
    # not using the variable. But the key will be the variable _value_ 
    # so you still have a $ in there.
    # e.g. array[1]
    unset 'dirs[$index]'
  fi
    
  # Incrementing a variable either requires arithmetic expansion with 
  # double parenthesis, or using let
  let "index++"
done
All put together, you get this handy function that takes a parent directory and returns a menu to choose only subdirectories.
# Fetch a list of directories under the passed one
# @param[1] parent directory
# @returns DIRECTORY, directory path
function directoryPicker {
  # $1 = Parent Directory
  PARENT=$1
  cd $PARENT

  # make an array using a directory listing
  declare -a dirs=($(ls -d *))
    
  # We want to display only folders as options, 
  # so remove non-directories
  index=0
  for dir in "${dirs[@]}"; do
    if [ ! -d "${PARENT}/${dir}" ]; then
      # unset an array element by referencing the variable name, 
      # not using the variable. But the key will be the variable value
      # e.g. array[1]
      unset 'dirs[$index]'
    fi
    
    # Incrementing a variable either requires arithmetic expansion with 
    # double parenthesis, or using let
    let "index++"
  done

  # Then display the array as a numbered choice list
  select dir in "${dirs[@]}"; do
    DIRECTORY="${PARENT}/${dir}"
    break;
  done
}
To put this select menu into use, you can call this function from inside another, passing it a parent directory. This example will fetch all the updates for all the git repos in a subfolder. I have my repos checked out into groups based on the organization or user that owns them, or based on a theme they have in common. So there are folders for my employer, my personal stuff, friends, and themed project topics like “WordPress” or “cryptocurrency”. So passing my updateRepos function a menu lets me update only a specific set of repos at once.
# Update all the repos under a parent folder
function updateRepos {
  # Get a list of subfolders, using our chooser to show a list
  directoryPicker "/Users/$(whoami)/Developer/github.com"
  REPOBASE=$DIRECTORY

  # Like we did in the chooser, create an array of subfolders
  cd $REPOBASE
  declare -a repos=($(ls -d *))

  # for each repo, fetch/pull latest
  for repo in "${repos[@]}"; do
    # Only bother with directories that have a .git subfolder
    if [[ -d $REPOBASE/${repo} && -d $REPOBASE/${repo}/.git ]]; then
      echo "Checking ${repo} for updates"
      cd $REPOBASE/${repo}
      git fetch
      git pull
    fi
  done
}

Prompt for a timespan in shell scripts

If you want to capture user input in a shell script (I prefer bash! Do what you want, but this script is bash), you’re looking for the command read.

This version gives the user a default start date 30 days in the past and will adjust the end date to be 30 days after whatever they enter.

#!/bin/bash
# Prompt a user for start and end dates

# Initialize start date variable with a default that is *30 days* ago
start_date=$( date -v -30d +"%Y-%m-%d" )

# Prompt the user for the start date
echo "What start date should I use? (format: YYYY-mm-dd, default: $start_date)"
read start_date_in
if [ "$start_date_in" != "" ]; then
    # If the user entered a date, use it
    start_date=$start_date_in
fi

# Initialize the end date as *30 days* after the start date
end_date=$( date -j -u -f "%Y-%m-%d" -v +30d "${start_date}" +"%Y-%m-%d" )

# Prompt the user for the end date
echo "What end date should I use? (format: YYYY-mm-dd, default: $end_date)"
read end_date_in
if [ "$end_date_in" != "" ]; then
    # Again, if the user entered a date, use it
    end_date=$end_date_in
fi

# Display the date span so the user knows!
echo "Processing between $start_date and $end_date"

# If you need to, convert these dates to timestamp for computer-friendly biz.
start_date_timestamp=$( date -j -u -f "%Y-%m-%d" "${start_date}" +"%s" )
end_date_timestamp=$( date -j -u -f "%Y-%m-%d" "${end_date}" +"%s" )

Generate a list of urls from a sitemap

When I’m load testing a site, I like to get a list of urls to run against. There’s not much point in checking the home page constantly, let’s find some variety.

This script expects a sitemap or sitemap index, and will give you back a text file with urls.

There is a dependency on xmlstarlet, a command line program for dealing with XML files. If you’re using homebrew it is simple to install xmlstarlet with brew install xmlstarlet.

When using XML to deal with a sitemap, and the namespaced elements in one, bind the namespace to a prefix and prepend it to the name, like this

xmlstarlet sel -N x='http://www.sitemaps.org/schemas/sitemap/0.9'

Source: http://xmlstar.sourceforge.net/doc/UG/xmlstarlet-ug.html#idm47077139669232

function get_urls_from_sitemap {
    # $1 sitemap_index
    SITEMAP_INDEX=$1

    OUTPUT_FIlE=urls.txt
    # Reset the output file
    : > $OUTPUT_FIlE

    # We use the namespaced in a few places so plop it here
    XMLSCHEMA='http://www.sitemaps.org/schemas/sitemap/0.9'

    # Check we got an XML file first by checking the content type
    isXML=$(curl -sS -o sitemap_index.xml -w '%{content_type}' "$SITEMAP_INDEX")

    # If it is an XML file, let's go with it. 
        # We'll get errors if it isn't a sitemap anyway
    if [[ $isXML = *"text/xml"* ]]; then

        echo "Getting urls from index: $SITEMAP_INDEX"

        # Read the sitemap index
        xmlstarlet sel -N x=$XMLSCHEMA -t -v '//x:loc' -n <sitemap_index.xml > sitemaps.txt

        # Then loop through the results!
        exec 4< sitemaps.txt
        while read <&4 SITEMAP; do

            # Some of these are url encoded, just quietly fix that!
            SITEMAP_URL=$(echo "$SITEMAP" | sed "s/\&amp;/\&/g")

            # This is the same content type check from before
            isXML=$(curl -sS -o sitemap.txt -w '%{content_type}' $SITEMAP_URL)

            if [[ $isXML = *"text/xml"* ]]; then
                # If this is an XML file, get more urls from it!
                echo "Getting urls from sitemap: $SITEMAP_URL"
                xmlstarlet sel -N x=$XMLSCHEMA -t -v '//x:loc' -n <sitemap.txt >> $OUTPUT_FIlE
            else
                # Just add non XML to the urls file
                echo $SITEMAP_URL >> $OUTPUT_FIlE
            fi

            rm -f sitemap.txt
        done

        rm -f sitemaps.txt
    else
        echo "Yo, this isn't an XML file"
    fi

    rm -f sitemap_index.xml
}