The Sourcery: TextFileConversion

[tyche@ctc tyche]$ perl -npe 's/\r$//' <test >test2

[tyche@ctc tyche]$ cat -v test2
#!/bin/csh
echo "hello"

[tyche@ctc tyche]$ ./test2
bash: ./test2.sh: Permission denied

[tyche@ctc tyche]$ chmod u+x test2

[tyche@ctc tyche]$ ./test2
hello

[tyche@ctc tyche]$

DOS -> UNIX: ruby -p -i.bak -e '$_.sub!(/\r$/,"")' file_name
DOS -> Mac: ruby -n -i.bak -e 'print $_.chomp+"\r"'

UNIX -> DOS: ruby -p -i.bak -e '$_.sub!(/$/,"\r")' file_name
UNIX -> Mac: ruby -n -i.bak -e 'print $_.chomp+"\r"'

Mac -> UNIX: ruby -p -i.bak -e '$_.gsub!(/\r/,"\n")' file_name
Mac -> DOS: ruby -p -i.bak -e '$_.gsub!(/\r/,"\r\n")'

Help! My script won't execute.

[tyche@ctc tyche]$ ./test
bash: ./test: No such file or directory

It's obviously there.

[tyche@ctc tyche]$ ls -la test
-rwxr--r--    1 tyche tyche       25 Apr 13 07:17 test

Permissions are correct too.

[tyche@ctc tyche]$ cat test
#!/bin/csh
echo "hello"

Looks fine.

[tyche@ctc tyche]$ which csh
/bin/csh

I also have cshell or tcshell installed and the path is correct on my shebang line.

[tyche@ctc tyche]$ ./test
bash: ./test: No such file or directory

So what could it be?

Let's look at the actual contents.

[tyche@ctc tyche]$ cat -v test
#!/bin/csh^M
echo "hello"^M

I didn't put those funny ^M thingies in there. I'll just remove them.

[tyche@ctc tyche]$ notepad test
bash: notepad: command not found

On Linux you say to yourself, "Self, where's the flaming editor at? Oh doh! I'm not on Windows anymore. I know. I'll just ftp the file down to my Windows machine, edit it and upload it. Yeah, that's what I'll do.".

Or Cygwin you are in the flaming notepad editor and can't see any ^M thingies.

Well the ^M thingies are called Carriage returns.

The EOL or end of line sequence on DOS/Windows?/OS2 is CRLF or carriage return followed by linefeed, or in C parlance, "\r\n", or in ASCII (015)(012).

The EOL used by Unix-based systems is LF, "\n" or (012).

Luckily most well written Unix tools recognize ANSI X34 standards for EOL and handle the \r\n EOL sequence (i.e. gcc, make, etc.). Some don't. In particular, bash, cshell, and tcshell do not, well especially when found on the shebang line; #!/bin/csh^M causes the shell to search for the program named "csh\r".

Rule Number One

Don't use notepad. Don't even use it on Window's files. It will even mess up application configuration files for Windows apps that happen to look like text, but contain special characters (i.e. tabs).'''

You can get away with using wordpad (write) as long as you remember to save the file as text and not rtf format, and don't do tab to space conversions.

Use either a unix editor or use textpad, ultraedit or another windows editor that recognizes and preserves EOL sequences, and other control characters.

Okay so how do I convert them back...

If you have perl do this:

[tyche@ctc tyche]$ perl -npe 's/\r$//' <test >test2

[tyche@ctc tyche]$ cat -v test2
#!/bin/csh
echo "hello"

[tyche@ctc tyche]$ ./test2
bash: ./test2.sh: Permission denied

[tyche@ctc tyche]$ chmod u+x test2

[tyche@ctc tyche]$ ./test2
hello

[tyche@ctc tyche]$

Note do not try redirecting input and output to the same file. It'll magically disappear. There's a way of getting around this but I forget it.

Darklord says here's the way..

[tyche@ctc tyche]$ perl -pi~ -e 's/^M//g' *.c
 make sure to use ctrl v to get ^ and then ctrl m for M else it wont work

Perl will make a backup of each file processed ending in ~.

If you don't have perl, use sed.

[tyche@ctc tyche]$ sed -e 's/.$//' <test >test2

Careful with that one above as it assumes all lines end in CRLF and strips off the last character.

If you use vim or vi you can edit a DOS/Windows? format file and force a change to the save format with the command ":se ff=unix"

If you have cygutils installed on cygwin, and I do recommend it, you've got dos2unix, unix2dos, d2u, u2d.

[tyche@ctc tyche]$ d2u test

Conversion in place. Quite handy. I believe some Linux distros now ship dos2unix or conv

[tyche@ctc tyche]$ conv -d2u test

Finally FTP supports automatic CRLF to LF and LF to CRLF conversion by using the 'ascii' command to set the transfer mode before uploading or downloading a file. One uses 'binary' to get back to binary mode transfer. One warning though, don't forget switch back to 'binary' mode before ftp-ing binaries like zips, exes, tars and gzipped files around, as they will be mercilessly destroyed in transfer.

More ways... If you have ruby installed you can do the following one liners:

DOS -> UNIX: ruby -p -i.bak -e '$_.sub!(/\r$/,"")' file_name
DOS -> Mac:  ruby -n -i.bak -e 'print $_.chomp+"\r"'

UNIX -> DOS: ruby -p -i.bak -e '$_.sub!(/$/,"\r")' file_name
UNIX -> Mac: ruby -n -i.bak -e 'print $_.chomp+"\r"'

Mac -> UNIX: ruby -p -i.bak -e '$_.gsub!(/\r/,"\n")' file_name
Mac -> DOS:  ruby -p -i.bak -e '$_.gsub!(/\r/,"\r\n")'