History of TextFileConversion

TheSourcery | RecentChanges | Preferences | Index | RSS

Difference (from prior major revision) (no other diffs)

Changed: 1,149c1,145

Help! My script won't execute.


 [tyche@ctc tyche]$ ./test

 bash: ./test: No such file or directory

It's obviously there.


 [tyche@ctc tyche]$ ls -la test

 -rwxr--r--    1 tyche tyche       25 Apr 13 07:17 test

Permissions are correct too.


 [tyche@ctc tyche]$ cat test

 #!/bin/csh

 echo "hello"

Looks fine.


 [tyche@ctc tyche]$ which csh

 /bin/csh

I also have cshell or tcshell installed and the path is correct on my shebang line.


 [tyche@ctc tyche]$ ./test

 bash: ./test: No such file or directory

So what could it be?

Let's look at the actual contents.


 [tyche@ctc tyche]$ cat -v test

 #!/bin/csh^M

 echo "hello"^M

I didn't put those funny ^M thingies in there. I'll just remove them.


 [tyche@ctc tyche]$ notepad test

 bash: notepad: command not found

On Linux you say to yourself, "Self, where's the flaming editor at? Oh doh! I'm not on Windows anymore. I know. I'll just ftp the file down to my Windows machine, edit it and upload it. Yeah, that's what I'll do.".

Or Cygwin you are in the flaming notepad editor and can't see any ^M thingies.

Well the ^M thingies are called Carriage returns.

The EOL or end of line sequence on DOS/Windows?/OS2 is CRLF or carriage return
followed by linefeed, or in C parlance, "\r\n", or in ASCII (015)(012).
The EOL used by Unix-based systems is LF, "\n" or (012).

Luckily most well written Unix tools recognize ANSI X34 standards for EOL and
handle the \r\n EOL sequence (i.e. gcc, make, etc.). Some don't. In particular, bash, cshell,
and tcshell do not, well especially when found on the shebang line; #!/bin/csh^M causes
the shell to search for the program named "csh\r".

Rule Number One

Don't use notepad. Don't even use it on Window's files. It will even
mess up application configuration files for Windows apps that happen
to look like text, but contain special characters (i.e. tabs).'''

You can get away with using wordpad (write) as long as you remember to save
the file as text and not rtf format, and don't do tab to space conversions.

Use either a unix editor or use textpad, ultraedit or another windows
editor that recognizes and preserves EOL sequences, and other control
characters.

Okay so how do I convert them back...

If you have perl do this:


 [tyche@ctc tyche]$ perl -npe 's/\r$//' <test >test2

 

 [tyche@ctc tyche]$ cat -v test2

 #!/bin/csh

 echo "hello"

 

 [tyche@ctc tyche]$ ./test2

 bash: ./test2.sh: Permission denied

 

 [tyche@ctc tyche]$ chmod u+x test2

 

 [tyche@ctc tyche]$ ./test2

 hello

 

 [tyche@ctc tyche]$

Note do not try redirecting input and output to the same file. It'll
magically disappear. There's a way of getting around this but I forget it.

Darklord says here's the way..


 [tyche@ctc tyche]$ perl -pi~ -e 's/^M//g' *.c

  make sure to use ctrl v to get ^ and then ctrl m for M else it wont work

Perl will make a backup of each file processed ending in ~.

If you don't have perl, use sed.


 [tyche@ctc tyche]$ sed -e 's/.$//' <test >test2

Careful with that one above as it assumes all lines end in CRLF and
strips off the last character.

If you use vim or vi you can edit a DOS/Windows? format file and force
a change to the save format with the command ":se ff=unix"

If you have cygutils installed on cygwin, and I do recommend it,
you've got dos2unix, unix2dos, d2u, u2d.


 [tyche@ctc tyche]$ d2u test

Conversion in place. Quite handy. I believe some Linux distros
now ship dos2unix or conv


 [tyche@ctc tyche]$ conv -d2u test

Finally FTP supports automatic CRLF to LF and LF to CRLF conversion
by using the 'ascii' command to set the transfer mode before uploading or downloading
a file. One uses 'binary' to get back to binary mode transfer.
One warning though, don't forget switch back to 'binary' mode before ftp-ing binaries
like zips, exes, tars and gzipped files around, as they will be mercilessly destroyed in
transfer.

Help! My script won't execute.


 [tyche@ctc tyche]$ ./test
 bash: ./test: No such file or directory

It's obviously there.


 [tyche@ctc tyche]$ ls -la test
 -rwxr--r--    1 tyche tyche       25 Apr 13 07:17 test

Permissions are correct too.


 [tyche@ctc tyche]$ cat test
 #!/bin/csh
 echo "hello"

Looks fine.


 [tyche@ctc tyche]$ which csh
 /bin/csh

I also have cshell or tcshell installed and the path is correct on my shebang line.


 [tyche@ctc tyche]$ ./test
 bash: ./test: No such file or directory

So what could it be?

Let's look at the actual contents.


 [tyche@ctc tyche]$ cat -v test
 #!/bin/csh^M
 echo "hello"^M

I didn't put those funny ^M thingies in there. I'll just remove them.


 [tyche@ctc tyche]$ notepad test
 bash: notepad: command not found

On Linux you say to yourself, "Self, where's the flaming editor at? Oh doh! I'm not on Windows anymore. I know. I'll just ftp the file down to my Windows machine, edit it and upload it. Yeah, that's what I'll do.".

Or Cygwin you are in the flaming notepad editor and can't see any ^M thingies.

Well the ^M thingies are called Carriage returns.

The EOL or end of line sequence on DOS/Windows?/OS2 is CRLF or carriage return followed by linefeed, or in C parlance, "\r\n", or in ASCII (015)(012).

The EOL used by Unix-based systems is LF, "\n" or (012).

Luckily most well written Unix tools recognize ANSI X34 standards for EOL and handle the \r\n EOL sequence (i.e. gcc, make, etc.). Some don't. In particular, bash, cshell, and tcshell do not, well especially when found on the shebang line; #!/bin/csh^M causes the shell to search for the program named "csh\r".

Rule Number One

Don't use notepad. Don't even use it on Window's files. It will even mess up application configuration files for Windows apps that happen to look like text, but contain special characters (i.e. tabs).'''

You can get away with using wordpad (write) as long as you remember to save the file as text and not rtf format, and don't do tab to space conversions.

Use either a unix editor or use textpad, ultraedit or another windows editor that recognizes and preserves EOL sequences, and other control characters.

Okay so how do I convert them back...

If you have perl do this:


 [tyche@ctc tyche]$ perl -npe 's/\r$//' <test >test2
 
 [tyche@ctc tyche]$ cat -v test2
 #!/bin/csh
 echo "hello"
 
 [tyche@ctc tyche]$ ./test2
 bash: ./test2.sh: Permission denied
 
 [tyche@ctc tyche]$ chmod u+x test2
 
 [tyche@ctc tyche]$ ./test2
 hello
 
 [tyche@ctc tyche]$

Note do not try redirecting input and output to the same file. It'll magically disappear. There's a way of getting around this but I forget it.

Darklord says here's the way..


 [tyche@ctc tyche]$ perl -pi~ -e 's/^M//g' *.c
  make sure to use ctrl v to get ^ and then ctrl m for M else it wont work

Perl will make a backup of each file processed ending in ~.

If you don't have perl, use sed.


 [tyche@ctc tyche]$ sed -e 's/.$//' <test >test2

Careful with that one above as it assumes all lines end in CRLF and strips off the last character.

If you use vim or vi you can edit a DOS/Windows? format file and force a change to the save format with the command ":se ff=unix"

If you have cygutils installed on cygwin, and I do recommend it, you've got dos2unix, unix2dos, d2u, u2d.


 [tyche@ctc tyche]$ d2u test

Conversion in place. Quite handy. I believe some Linux distros now ship dos2unix or conv


 [tyche@ctc tyche]$ conv -d2u test

Finally FTP supports automatic CRLF to LF and LF to CRLF conversion by using the 'ascii' command to set the transfer mode before uploading or downloading a file. One uses 'binary' to get back to binary mode transfer. One warning though, don't forget switch back to 'binary' mode before ftp-ing binaries like zips, exes, tars and gzipped files around, as they will be mercilessly destroyed in transfer.

More ways... If you have ruby installed you can do the following one liners:


 DOS -> UNIX: ruby -p -i.bak -e '$_.sub!(/\r$/,"")' file_name
 DOS -> Mac:  ruby -n -i.bak -e 'print $_.chomp+"\r"'
 
 UNIX -> DOS: ruby -p -i.bak -e '$_.sub!(/$/,"\r")' file_name
 UNIX -> Mac: ruby -n -i.bak -e 'print $_.chomp+"\r"'
 
 Mac -> UNIX: ruby -p -i.bak -e '$_.gsub!(/\r/,"\n")' file_name
 Mac -> DOS:  ruby -p -i.bak -e '$_.gsub!(/\r/,"\r\n")'

All material on this Wiki is the property of the contributing authors.
©2004-2006 by the contributing authors.
Ideas, requests, problems regarding this site? Send feedback.

	Revision 9 . . August 14, 2005 5:24 am by JonLambert [cleaned up spacing and added ribuy one-liners]
	Revision 8 . . December 1, 2004 2:14 am by JonLambert