Linux Scripting Help Needed

Talk about whatever topic you'd like, RPG related or not. (But please discuss things related to our software in the Tools section, below.)

Moderators: dorpond, trevor, Azhrei

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Linux Scripting Help Needed

Post by Steel Rat »

Hello All,

I need some help creating a Linux script that will run as a cron. Basically I need it to delete files in a specific folder that are more than two days old.

Anyone able to help with this?

Thanks!
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
giliath
RPTools Founder
Posts: 275
Joined: Tue Jan 31, 2006 11:10 am
Location: Austin, TX

Post by giliath »

Here is a crontab entry I have.

Code: Select all

0 0 * * * /usr/bin/find /u01/oradata/logs -mtime +15 -exec rm {} \;
At midnight every day, find files older than 15 days and remove them.
~Giliath

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

giliath wrote:Here is a crontab entry I have.

Code: Select all

0 0 * * * /usr/bin/find /u01/oradata/logs -mtime +15 -exec rm {} \;
At midnight every day, find files older than 15 days and remove them.
Thanks! I'll give that a try.
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Post by Azhrei »

I would recommend two changes:

Code: Select all

0 0 * * * find /some/dir/path -mtime +2 -print | xargs rm
The first change is the use of "+2" instead of "+15", since you actually said you were looking for files older than two days. :)

The second item is related to the rm command. This is difficult to explain, but the -exec option on find requires a trailing ; as a terminator. (Never mind that the end-of-line should be a valid terminator -- it's not.) Anyway, using a backslash in front of the semicolon is not a portable choice of syntax. It's better to put single quotes around the semicolon instead.

But an even better solution (both in syntax and in execution performance) is to run the output of find into the xargs command. The xargs command will read in a bunch of filenames and execute a single rm for all filenames read in. This results in a huge performance gain. Consider 100 filenames to be removed. The find command would have to fork() and exec() each rm command separately. With xargs, up to all 100 filenames will be read in and a single rm command will be run. (The number of filenames that can be passed to rm is system-dependent and may be as low as 1 or as high as the command line length limit will allow, which on Linux is 1MB.) There are other reasons for preferring the use of xargs as well, such as parallelism, but that's for another day. :)

Anyway, the original solution will work. My recommendation increases portability and, hence, reliability. Your mileage may vary. :)

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

Thanks Azhrei. I did change the +15 to +2. I'll see how things go for a few days and make the other changes if needed.
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

Hey Azhrei,

I'm getting an error when the following executes:

find /home/rpgmap/g2data/tmp -mtime +2 -print | xargs rm

The error is: rm: too few arguments
Try `rm --help' for more information.

I tried adding a -r for recursive (which I need since there are folders in the path that will need deleting too.) but it didn't like that either.

Thanks!
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Post by Azhrei »

Steel Rat wrote:Hey Azhrei,

I'm getting an error when the following executes:

find /home/rpgmap/g2data/tmp -mtime +2 -print | xargs rm

The error is: rm: too few arguments
Try `rm --help' for more information.

I tried adding a -r for recursive (which I need since there are folders in the path that will need deleting too.) but it didn't like that either.

Thanks!
The find command will descend through all directory levels, so you don't want rm -r or a directory could be selected for removal when the files inside the directory aren't old enough yet! Don't do that!

The error itself is harmless (it's a byproduct of the find command not producing any output), so there is an easy way to avoid it. Send all error messages to /dev/null using:

find /home/rpgmap/g2data/tmp -mtime +2 -print | xargs rm 2>/dev/null

The other issue that you mentioned is that the directory structure contains subdirectories. I'm guessing by your statement that you want empty directories that are also old directories deleted as well. The find command you have will not do that because rm won't delete directories without the -r option. So change your find command into two commands:

find /home/rpgmap/g2data/tmp ! -type d -mtime +2 -print | xargs rm 2>/dev/null
find /home/rpgmap/g2data/tmp -type d -depth -print | xargs rmdir 2>/dev/null

The first command is the one that deletes files. Do this one first. The second one deletes directories older than two days, but does so "depth first", meaning subdirectories are done before parent directories. It's important to delete the files first so that the directory will be empty when the find command gets to it. Unfortunately, deleting a file from a directory causes the timestamp on the directory to be updated! So I removed the -mtime +2 option. This means that every directory will be given to the rmdir command (via xargs), but rmdir will only delete empty directories, so that's not a problem.

There are more efficient ways of handling this. For example, in Perl I would walk the directory structure using the File::Find module and generate a list of everything that needs to be done. Then I'd go back in a separate pass and actually do it. That way the modification time on the directory could still be used.

Btw, since you now have two commands that you want to run sequentially, if you're putting these into a crontab file, don't use two separate lines. Instead, put both commands on one line and separate them with a semicolon (";"). This forces them to run sequentially.

Gotta run, but I'll check back here again later today. :)

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

I ended up using Giliath's example and added -dfr to the rm command.

I didn't see that any directories got the timestamp updated and therefore not deleted, but will have to see again tonight when it runs.
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Post by Azhrei »

Okay. Just to be sure: the rm -r command will delete any directory it is given along with ALL files and ALL subdirectories. If the directory is older than 2 days, but the files are not, the directory will still be deleted.

(This might not be possible in your application; directory timestamps are updated when the directory is changed: file created, file deleted, file renamed. Modifying an existing file does NOT update the timestamp on the directory.)

Just be sure that's what you want. :)

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

It is. It's for RPGMapShare. Sometimes very large shopping carts aren't deleted (we're talking up to half a gig or more), which quickly fills up the drive. So I do want all files and dirs deleted which are more than 2 days old. Right now I have to do this manually, and if I forget, I start getting disk full reminders after a couple weeks.
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

I really feel like a linux dummy here.

I've moved to a new server as of a couple months ago, and my old crontab didn't carry over.

So I'm trying to re-enter the cron I was using:

Code: Select all

0 0 * * * /usr/bin/find /home/rpgmap/g2data/tmp -mtime +2 -exec rm -dfr {} \;
But can't for the life of me figure out how to edit the crontab file. I run crontab -e and it comes up, but won't let me insert a line at the bottom or anything, just beeps at me. It's frustrating. Any ideas?
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Steel Rat
Great Wyrm
Posts: 1765
Joined: Tue Jun 13, 2006 5:55 pm
Location: Oak Harbor, WA
Contact:

Post by Steel Rat »

Never mind. I finally found a site that told how to change the default editor for my OS. So I changed it to Pico. which has an actual command menu that's visible. Made the changes, verified, done.
Steel Rat
-----------
RPGMapShare.com - RPG Maps and Mapping objects.
Discord Server

User avatar
Orchard
Great Wyrm
Posts: 1852
Joined: Fri May 09, 2008 10:45 am
Location: Doylestown PA
Contact:

Post by Orchard »

Steel Rat wrote:Never mind. I finally found a site that told how to change the default editor for my OS. So I changed it to Pico. which has an actual command menu that's visible. Made the changes, verified, done.
Ah, pico. I love pico. Pico is such a great little editor.

The other editor of choice for me when on a command line is nano, which is (not surprisingly) essentially the same program. WHY would anyone use the arcane mess that is vi or vim?
0+0=1, for very unstable CPUs.

User avatar
Mr. Pokeylope
Giant
Posts: 118
Joined: Mon Aug 11, 2008 9:24 pm

Post by Mr. Pokeylope »

Orchard wrote:The other editor of choice for me when on a command line is nano, which is (not surprisingly) essentially the same program. WHY would anyone use the arcane mess that is vi or vim?
You don't want to start this.

User avatar
Orchard
Great Wyrm
Posts: 1852
Joined: Fri May 09, 2008 10:45 am
Location: Doylestown PA
Contact:

Post by Orchard »

Mr. Pokeylope wrote:
Orchard wrote:The other editor of choice for me when on a command line is nano, which is (not surprisingly) essentially the same program. WHY would anyone use the arcane mess that is vi or vim?
You don't want to start this.
I've already started it. There's a relevant xkcd comic about this topic, which I will pre-emptively post to head off any further comments.

Image

The truth of the matter is quite simple though; emacs, vi, vim and other context sensitive text editors from unix are rather arcane, and you needed to learn a lot of very unintuitive commands to be able to accomplish the simplest of tasks with very minimal user feedback. I am very glad that more userfriendly programs exist, and see no reason to go back to something more arcane.
0+0=1, for very unstable CPUs.

Post Reply

Return to “General Discussion”