Expert shell scripting, Study Guides, Projects, Research of Operating Systems

Download Expert shell scripting and more Study Guides, Projects, Research Operating Systems in PDF only on Docsity!

Expert Shell Scripting

Ron Peters

iv

Contents at a Glance

About the Author............................................................... xv

About the Technical Reviewer................................................... .xvii

Acknowledgments.............................................................. xix

Introduction................................................................... xxi

PART 1 ■ ■ ■ Basic Scripting Techniques

■ CHAPTER 1 Shell Script Debugging........................................ 3 ■ CHAPTER 2 Standard Function Library.................................... 13 ■ CHAPTER 3 Date and Time Manipulation.................................. 19 ■ CHAPTER 4 Comparisons and Tests...................................... 25 ■ CHAPTER 5 Accepting Command-Line Options, Switches,

and Parameters.............................................. 31

■ CHAPTER 6 Testing Variables and Assigning Defaults...................... 37 ■ CHAPTER 7 Indirect Reference Variables.................................. 43 ■ CHAPTER 8 Shell Process Tree........................................... 49 ■ CHAPTER 9 Data Redirection............................................. 59 ■ CHAPTER 10 Piping Input to read.......................................... 65

PART 2 ■ ■ ■ System Interaction and Advanced

Techniques

■ CHAPTER 11 Math from the Shell.......................................... 75 ■ CHAPTER 12 cron........................................................ 81 ■ CHAPTER 13 Self-Linked Scripts........................................... 87 ■ CHAPTER 14 Throttling Parallel Processes................................. 91 ■ CHAPTER 15 Command-Line Editing and History............................ 99 ■ CHAPTER 16 Scripting from the Command Line............................ 103 ■ CHAPTER 17 Automating User Input with expect........................... 107

PART 3 ■ ■ ■ Useful Scripts

viii ■C O N T E N T S

PART 2 ■ ■ ■ System Interaction and Advanced

xv

About the Author

■ RON PETERS has worked as a system administrator for most of the last 15 years. He was a senior administrator at Intel in a 24/ production environment and was the primary administrator of a large compute cluster dedicated to design work. He is now a Linux/ UNIX administrator for Columbia Sportswear. He enjoys spending time with his family, restoring his Dodge Challenger, and playing racquetball.

xvii

About the Technical Reviewer

■ BRIAN CULP has worked professionally in the information-services industry for 20 years. Throughout those years he has worked with startups and small businesses, and spent a dozen years employed by a leading company in the IT industry. Brian has spent time on service desks, as a UNIX systems admin, a project manager, an e-commerce/business web site administrator, and a solutions developer. Brian’s development and use of UNIX shell scripts has always grown out of specific needs, as he always seems to be in some stage of trying to solve a technical problem. He hopes you will find the scripts and methods described in this book useful in building your own problem-solving toolkit.

xxi

Introduction

I learned the basics of programming when I was in school; I learned how to shell-script by example. I’ve met and worked with many system administrators and other *NIX folks, each of whom has their own bag of tricks when it comes to managing a system, interacting with their environment, or coding a script. It’s always very useful to have conversations and interact with people like this because you invariably gain some tidbits that you can throw into your own collection of tricks. I decided to collect all the useful shell-scripting and interaction techniques I have learned through the years and combine them into one beneficial reference guide. In fact, I used some of my own notes about those techniques while writing this book. Since I haven’t memorized everything present in this book, I would periodically look up items when I was working on various tasks. I want this book to be the beginning of a higher-level reference library that can be added to and can grow continually. You might be aware of the large number of shell-scripting books and online resources aiding in the mastery of shell scripting. Many are excellent and cover a wide range of topics. The main purpose of this book is to combine some of the most unique tools, code snippets, and scripts that go beyond the level of basic scripts. I wanted to create a cookbook of sorts— lesser-known recipes and fairly advanced algorithms that have proved useful to me. I have included scripts you can use as is, and sample scripts illustrating a specific algo- rithm. I also demonstrate a few complex commands that may be useful on the command line. I have tried to tailor the scripts to be useful at multiple levels. Most times, however, there is little or no error-checking since that is not necessarily the point of a specific script. You must be prepared to make modifications to fit your local environment.

How This Book Came About

My friend Brian Culp and I have worked together as UNIX system administrators for many years. Periodically, Brian or I will be working on some script and run into a problem. One of us will stop, walk over to the other, and say something like, “Do you have any code that does X ?” The answer may be no, in which case we’ll launch into a discussion on how we might tackle the problem, or come up with a few different solutions. However, many times it might be something like, “Hmm, yeah, I think I remember doing something like that in a script that does X on system Y. Let me look for a minute.” A few carefully chosen grep commands, and the solution is at hand.

xxii ■I N T R O D U C T I O N

Although finding the solution we want is great, it’s not the most efficient use of our time. To go from having a place to store and organize all of our (and, of course, other programmers’) gems and having them in a heavily documented form, to writing a book on the subject was only a short step. Even though it is possible to search online references quickly for specific code, there were many occasions when I just wanted to pull a book off my shelf. It’s not always obvious what to search for when you have a specific itch that needs scratching, so you’re not sure exactly to what search for online. This is to some extent an expression of my own limitations: my family and friends think I’m a computer guru, but rest assured, I know better. There are many programmers out there who are much more adept at shell coding than I am. I mainly intended to collect, order, and explain code that I have found to be highly useful in my professional experi- ence as a system administrator, and share that information with others.

Who Should Read This Book

The book is meant for the intermediate shell coder up to the advanced shell-code hacker, because I don’t explain many basic programming structures. If you’re looking for that type of book, you should look to the resources mentioned in Appendix C. This is not to say that the beginner won’t find this book useful; it may work well as a supple- mentary reference to a more traditional shell-scripting training guide. But there is a difference between learning English as a second language and learning how to apply sarcasm. This book is like sarcasm in that example; it assumes some basic shell-code literacy. I go into great detail about how and why the scripts were written in their present form, and I include some explanation of how to avoid certain problems. Much of my learning came from sources heavy in obfuscation and light on clarity, so I tried to be as explicit as possible, and favored explaining too much rather than too little. You can think of many chapters I included as shell scripts with extremely detailed commentary. The book is divided into three parts: “Basic Scripting Techniques,” “System Interaction and Advanced Techniques,” and “Useful Scripts.” Most chapters serve as stand-alone discussions, although they may refer to other chapters on some minor points.

Issues and Ideas

I have made every effort to test the code that I included in this book to validate that it works. With a project of this size, however, even with the number of eyes that have reviewed it, there may be mistakes. I would like to know about the mistakes as well as, and more importantly, any other ideas and scripts that could be used for future revisions of this book. Please drop me a note at rbpeters@peterro.com.

3

■ ■ ■

C H A P T E R 1

Shell Script Debugging

E ven though this book isn’t a “how to script” manual, some concepts that are funda- mental to writing successful scripts should be discussed. Debugging is one of them. Debugging code is a significant part of writing code. No matter how disciplined you are or how skilled you become at coding, you will have bugs in your code, in the form of either syntax or logic errors. The syntactical problems tend to be simpler to resolve since many times they show up when the code throws an error when it is run. The logical bugs, on the other hand, may be more difficult to track down since the code may run without error, but the resulting output does not match the design of the program. The more complex your code becomes as your skill increases, the more difficult these types of problems will be to detect. Since writing bug-free code is nearly impossible, you need a few techniques up your sleeve that will help you finish, diagnose, repair, and clean up your code. This chapter pre- sents a few ways to debug code that I have used consistently and that help me extract details from the inner workings of my scripts. These techniques validate that the code is living up to my expectations and demonstrate where the code needs more work to per- form the intended task.

Shell Trace Options

The first technique—using the set command—is the simplest to implement and can give you great amounts of detail about how the logic is progressing and the values of variables internal to your script. Using the set command is really just using shell options to display verbose output when the script is running. One of the functions of the set command is to turn on and off the various options that are available in the shell. In this case, the option being set is -x, or xtrace. This is where the running script will, in addition to any normal output, display the expanded commands and variables of a given line of code before the code is run. With this increased output, you can easily view what is happening in the run- ning script and possibly determine where your problem lies. When you put the instruction set -x into your script, each of the commands that exe- cute after that set instruction will be displayed, together with any arguments that were supplied to the command, including variables and their values. Each line of output will be

4 C H A P T E R 1 ■ S H E L L S C R I P T D E B U G G I N G

preceded by a plus-sign (+) prompt to designate it as part of the trace output. Traced commands from the running shell that are being executed in a subshell are denoted by a double plus sign (++). To demonstrate what the use of set -x can do for you, consider this script: #!/bin/sh #set -x echo -n "Can you write device drivers? " read answer answer=echo $answer | tr [a-z] [A-Z] if [ $answer = Y ] then echo "Wow, you must be very skilled" else echo "Neither can I, I'm just an example shell script" fi Note that the set -x line is currently commented out. When this script is entered in the file example and run, the behavior is as expected. $ ./example Can you write device drivers? y Wow, you must be very skilled or $ ./example Can you write device drivers? n Neither can I, Im just an example shell script This is the output when the set -x line is uncommented: $ ./example

• echo -n 'Can you write device drivers? ' Can you write device drivers? + read answer y ++ tr '[a-z]' '[A-Z]' ++ echo y
• answer=Y
• '[' Y = Y ']'
• echo Wow, you must be very skilled Wow, you must be very skilled or